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FOREWORD 


The mission of the U. S. Army Research Institute for the Behavioral and Social Sciences (ARI) is 
to maximize individual and unit performance and readiness to meet the full range of world-wide Army 
missions through advances in the behavioral and social sciences. This investigation illuminates a 
previously unexplored aspect of Sternberg’s theory of intelligence by outlining the cognitive processes 
involved in acquiring tacit knowledge from experience, the problem solving skills involved in solving 
complex practical problems, and the interaction between knowledge and problem solving in expert 
performance. The reflection methods and case study assessments developed for this investigation can be 
used for further research in this area of inquiry. The reflection methods developed in this project can be 
directly applied to leadership development in the Army at multiple levels of leadership. A manual of 
materials was created for this purpose and the methods featured in the manual can also be adapted to 
specific training situations. The next step will be to verify that the methods in the manual work in actual 
training/leader development applications. 


MICHELLE SAMS 
Technical Director 
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DEVELOPING EFFECTIVE MILITARY LEADERS: FACILITATING THE ACQUISITION 
OF EXPERIENCE-BASED, TACIT KNOWLEGE 


EXECUTIVE SUMMARY _ 

Research Requirement: 

To develop and test methods to enhance practical problem solving skills and the ability to 
acquire experience-based (tacit) knowledge for purposes of application to leadership 
development and deepening knowledge of underlying cognitive processes. 

Procedure: 

Two related experimental studies were conducted to investigate the effect of theory-based 
reflection methods on practical problem solving in Army officer and college student samples 
using domain-specific and context-dependent measures to assess tacit knowledge in each sample. 
Two new case study scenario assessments were developed for each sample. Three theory-based 
reflection methods were designed according to the theory of practical intelligence. In a single 
three-hour session, participants were administered self-paced tacit knowledge pre-test 
assessments, covariate measures, an experimental reflection method or control condition, and 
tacit knowledge post-test assessments. Statistical analyses were conducted to test the effect of 
each experimental reflection method on tacit knowledge post-test performance. 

Findings: 

The skills underlying the acquisition of tacit knowledge can be taught. Reflection 
methods derived from the theory of practical intelligence effectively promoted practical problem 
solving in research participants. Theory-based reflection appears to be more effective than 
simple practice, as substantiated by the military leadership research (Research Effort 1), and 
certain types of reflection methods are more effective than others in promoting tacit knowledge 
acquisition. In the college life research (Research Effort 2), an analytic reflection control that 
prompted participants to examine and analyze domain-specific issues was less effective than a 
reflection method derived from the theory of practical intelligence. 

Utilization of Findings: 

This investigation showed that even very brief reflection methods based on the theory of 
practical intelligence can improve practical problem solving. However, all reflection methods are 
not alike. The effectiveness of a particular reflection method seems to depend upon the 
complexity of the task. The reflection methods and case study scenarios developed for this 
investigation can be directly applied for purposes of ongoing research and/or leadership 
development. 
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Developing Effective Military Leaders: 

Facilitating the Acquisition of Experience-Based, Tacit Knowledge 

INTRODUCTION 

The work environment of the future Army leader is characterized by an increased 
diversity in the nature of missions, a greater requirement for rapid, widely dispersed 
deployment, and a wider array of sophisticated technology—all of which require that 
officers more effectively and efficiently learn from experience. Research suggests that 
experience-based (tacit) knowledge is a critical component of success in military 
leadership (Horvath et al, 1999) and private sector management (Wagner & Sternberg, 
1991). A variety of reflection methods have been used to enhance or foster the exchange 
of tacit knowledge in a range of workplace settings (Argyris, 1994, 1999; Epstein, 1999; 
Watkins & Marsick, 1993). However, there appears to be limited empirical work to 
support the claim that experience-based learning can be facilitated by reflection on 
experience. Moreover, if reflection does indeed enhance one’s capacity to learn 
effectively from experience, it is unclear what types of reflection methods may be most 
effective. These questions were addressed in two field experiments, which tested the 
effect of reflection methods derived from Sternberg’s theory of practical intelligence 
(Sternberg et al., 2000) on practical problem solving in Army officers and college 
students. 

Prior research suggests that people differ in their ability to learn from experience 
(McCall, Lombardo, & Morrison, 1988; Sternberg, et al., 2000; Wagner & Sternberg, 
1991) and that acquired knowledge or skill is a product of the interaction between person 
and experience. Drawing from Sternberg’s (1985, 1997) theory of intelligence, which 
differentiates practical from analytical and creative abilities, the practical ability to 
acquire tacit knowledge and thereby learn from experience (i.e., practical intelligence) is 
itself a form of developing expertise. When previously acquired tacit knowledge is 
applied through action in response to new experience, the action tests the implications of 
that tacit knowledge. The information provided by the outcome may modify or 
complexify the cognitive network of domain-specific tacit knowledge. According to the 
theory, the ability to learn from experience develops into practical expertise through an 
iterative process of acquiring tacit knowledge, applying it to new situations, and 
developing the knowledge structure as a result of the outcome of actions taken in the 
context of experience. In this way, abilities are conceptualized as malleable processes as 
opposed to fixed traits. 

Reflection methods were developed based on Sternberg’s theoretical 
conceptualization of the structure of tacit knowledge and processes involved in 
developing it. Hypotheses and predictions were advanced consistent with Sternberg’s 
theory, in which practical intelligence is conceptualized as a form of developing expertise 
through a process of tacit knowledge acquisition, application, and development. 
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tacit knowledge in military leadership. According to Sternberg’s theory of practical 
intelligence, tacit knowledge is defined as a complex set of condition-action statements 
that are domain specific, and reflect “knowing how” versus “knowing that.” 

Tacit knowledge plays a key role in practical problem-solving and effective action 
(Sternberg et al., 2000). When practical problems are complex or ambiguous, such as 
those faced by military leaders in the current environment, practical skills and a flexible 
knowledge base are required to know when and how to take quick and effective action 
(Schon, 1983; Smith, Ford, & Kozlowski, 1997). Facilitating the acquisition of tacit 
knowledge improves upon an officer’s capacity to effectively isolate variables in the 
environment that are relevant to a particular problem, select actions that will lead to 
appropriate solutions, and implement these actions effectively (Schon, 1983; Tolman & 
Brunswik, 1935). As tacit knowledge is a critical component of practical intelligence, we 
expected to show that facilitating the acquisition of tacit knowledge would, in turn, 
improve practical problem solving. Improved practical problem solving has obvious 
implications for leadership effectiveness in the complex, rapidly changing environment 
of modem military service. 

Two complementary approaches to understanding tacit knowledge that have been 
explored in the literature were integrated to develop reflection methods for this research. 
The first approach, originated by Polanyi (1966) and examined more extensively by 
Sternberg and his colleagues (e.g., Sternberg & Wagner, 1992; Sternberg et al., 2000; 
Wagner, 1987; Wagner & Sternberg, 1991), is characterized by an information¬ 
processing explanation for how knowledge that is not easily articulated enhances 
performance. Efforts to facilitate the acquisition of tacit knowledge using this approach 
have targeted individuals’ ability to identify the appropriate environmental conditions on 
which to act (e.g., Sternberg, Wagner, & Okagaki, 1993). The second approach, 
originated by Schon (1983), focuses on developing tacit knowledge through personal 
reflection on the causal loop in which tacit knowledge, actions based on this knowledge, 
and consequences of the action taken, are situated. According to Schon (1983), action— 
which links tacit knowledge to action outcomes—and reflection—which links action 
outcomes to tacit knowledge—serve as the cornerstones of tacit knowledge development. 

Antonakis et al. (2001) incorporated both of these approaches in a model of tacit 
knowledge acquisition within the framework of Sternberg’s theory of practical 
intelligence and tacit knowledge. This model illustrates the dynamic interaction of 
condition and action aspects in the development of tacit knowledge. Sternberg’s 
theoretical treatment, which specifies cognitive and metacognitive processes involved in 
tacit knowledge acquisition and development, served as the basis for designing 
experimental reflection methods to enhance experience-based learning and to make 
predictions about their relative effectiveness. In this way, we attempted to determine the 
aspects of tacit knowledge on which attempts to facilitate such knowledge should focus 
in order to maximize both efficiency and effectiveness. 

In particular, three types of reflection methods were developed that draw attention 
to: 1) factors considered in problem identification (reflection on condition); 2) chosen 
action and action outcomes (reflection on action); and 3) a combination of both reflection 
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In particular, three types of reflection methods were developed that draw attention 
to: 1) factors considered in problem identification (reflection on condition); 2) chosen 
action and action outcomes (reflection on action); and 3) a combination of both reflection 
on condition and action. To measure the differential effect of these interventions on 
practical problem solving, two new military-specific case study instruments were 
developed to assess practical skills and tacit knowledge in military leadership. It was 
predicted that reflection methods that focus reflection on both the condition and action 
aspects of practical problem solving would improve scores on tacit knowledge measures 
more than methods that focus on one or the other aspect exclusively. It was also expected 
that reflection methods that focus reflection on both the condition and action aspect of 
practical problem solving would be superior to traditional analytical methods of 
reflection (e.g., assessing the advantages and disadvantages of a particular action in a 
given situation). 

Two studies were conducted to assess the relative effectiveness of experimental 
training interventions. The first effort involved the development of materials specific to 
the leadership in the U.S. Army and was administered to Army officers; these materials 
were designed to assess officers in multiple ranks and levels of command. However, this 
research effort was interrupted by the deployment of officers to the Iraq War. Therefore, 
we designed a second research effort that was adapted for administration to a sample of 
college students. It involved developing new measures of tacit knowledge in college life 
and adapting existing measures and interventions accordingly. To assist the Army, the 
reflection methods and case study instruments have been compiled in the form of a 
supplemental training manual that can be applied to a leadership development 
curriculum. 

BACKGROUND 

Inquiry into the nature and processes involved in learning from experience has 
been pursued by scholars from a wide range of disciplines including: learning theory 
(Dewey, 1938; Kolb, 1984), philosophy (Polanyi, 1966), psychology (Reber, 1989; Reber 
& Lewis, 1977; Sternberg et al., 2000; Wagner & Sternberg, 1985), and management 
(Nonaka & Takeuchi, 1995). Polanyi (1966) first proposed the importance of the ability 
to implicitly acquire tacit knowledge from experience and the critical role of attention. 
Neisser (1976) first made the distinction between practical intelligence as it pertains to 
learning from experience and intelligence associated with academic success. 

Broadly speaking, the investigation of tacit knowledge and its role in performance 
has been approached from two complementary perspectives. The first perspective stems 
from Polanyi’s (1966) recognition of individuals’ ability to implicitly acquire an 
understanding of the connection between a pattern of stimuli or events and an experience. 
This is illustrated by an example of a blind man identifying objects with a cane to show 
how patterns of stimulation to the hand that occur when the cane touches an object 
become tacitly understood as the presence of the object itself. He noted that the man 
using the cane would likely be unable to articulate how he “knows” there is an object at 
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Furthering Polanyi’s (1966) discussion of tacit knowledge, Sternberg and his 
colleagues (e.g., Sternberg & Wagner, 1992; Wagner, 1987; Wagner & Sternberg, 1991) 
developed and tested a theory of tacit knowledge and practical intelligence, a component 
of Sternberg’s broader theory of successful intelligence (Sternberg, 1985, 1997). The 
theory of successful intelligence recognizes the importance of analytical, creative, and 
practical abilities in the successful accomplishment of personally valued goals within a 
particular socio-cultural context. Investigations of tacit knowledge and practical 
intelligence based on this theory have explored cognitive information processes that 
occur when an individual acts on tacit knowledge in a particular situation. As discussed 
in more detail in the next section, Sternberg and his colleagues have described tacit 
knowledge as a complex set of condition-action mappings through which individuals 
select and execute the appropriate action (e.g., turning to avoid a wall) given specific 
environmental conditions (e.g., stimulation to the hand holding the cane). Extensive 
research has been devoted to outlining the nature and content of tacit knowledge, 
specifically the identification of the particular conditions to which action must be mapped 
in order to achieve desired action outcomes, as it is the condition calling for action that is 
the most difficult to articulate (see Sternberg et al., 2000, for a comprehensive review). 
Sternberg’s theory regarding tacit knowledge and practical intelligence has led to the 
development of occupation-specific (e.g., management, military leadership) tacit 
knowledge measures, in which individuals must select actions to solve practical problems 
based on their sensitivity to the environmental conditions in which the problems are 
situated. 

The second perspective through which tacit knowledge has been explored springs 
from Schon’s (1983) work, in which action is recognized as an important factor that 
shapes the contents of an individual’s tacit knowledge base. Schon’s emphasizes the role 
of action in shaping the content of tacit knowledge and unexpected action outcomes in 
leading an individual to reflect on tacit knowledge, or assumptions that led to the action. 
He examined different types of thought processes demonstrated by professionals, in 
particular those used to address novel problems in day-to-day work experience. Schon 
found that when customary responses prove ineffective, professionals engage in a natural 
form of reflection on-the-spot to try out and consider new action, which he calls 
reflection-in-action. This he distinguishes from reflection-on-action, which occurs 
retrospectively. Rather than specifying the precise conditions that give rise to particular 
actions, Schon’s work and that of several other investigators in the area of experience- 
based learning (e.g., Argyris, 1988; Epstein, 1999; Mezirow, 1991; Raelin, 1997) focuses 
on the hypothesis-testing process through which individuals come to develop or modify 
the tacit knowledge network through this process. Though time-consuming, reflection 
methods characteristic of this approach have been successfully used to enhance the 
performance of private sector managers (Argyris, 1994, 1999) and medical personnel 
(Epstein, 1999). 

These two approaches to understanding tacit knowledge differ in that the first 
approach, taken by Sternberg and his colleagues, focuses primarily on the content of tacit 
knowledge—the conditions giving rise to action—while the second approach, 
characterized by Schon’s work, focuses on the process of acquiring tacit knowledge—the 
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actions that are linked to particular conditions and how action outcomes inform the 
development of tacit knowledge. Using Polanyi’s example of the blind man, Sternberg’s 
approach focuses on the cognitive processes through which a pattern of stimulation to the 
hand gives rise to interpreting the experience of an object and the action that should be 
taken. In contrast, Schon’s approach focuses on how action, based on the expectation that 
a specific pattern of stimulation to the hand will result in a specific outcome, is tested. If 
an undesired action outcome results, reflection on the expectations that led to the action 
may serve to enhance the tacit knowledge that gave rise to the expectation. The two 
approaches are complementary in that both appear to describe parts of the same whole. 
That is, although it is critical that an individual recognize the environmental conditions 
indicating particular actions to take in a given situation, the individual must also 
recognize how that information relates to particular action outcomes and how reflecting 
on this relationship leads to enhanced tacit knowledge. 


Acquiring Tacit Knowledge 

Polanyi (1966) noted that a critical factor contributing to implicit, rather than 
explicit, acquisition of knowledge is the direction of attention away from the particular 
stimuli or events that give rise to an experience and toward the internal sensations 
stimulated by them. For example, people often report experiencing “butterflies in their 
stomach” when a potential romantic partner responds positively to a gesture of interest. 
Attention here is focused away from external cues such as the potential partner’s body 
language and toward the internal or visceral experience precipitated by the positive 
response. Further, if determinations of interest level are correct and a romantic 
involvement ensues, when asked, “How did you know she/he was interested?” a typical 
response might be, “I don’t know—I just got the right vibe,” rather than recognizing the 
specific cues or conditions on which their intuition was based. 

In order to improve such determinations and resulting actions, however, attention 
must be directed back to the particular body language that individuals used as cues for 
action. Working primarily in the domain of complex problem solving, Sternberg and his 
colleagues (Horvath et al., 1999; Sternberg et al., 2000; Sternberg & Wagner, 1992) have 
taken an information-processing approach to describe the relationship between the 
particular stimuli or events that give rise to action and the action itself. They describe 
tacit knowledge as a complex set of condition-action mappings through which 
individuals select and execute the appropriate action given specific environmental 
conditions. 

Sternberg and his colleagues characterize tacit knowledge as having three critical 
features. First, tacit knowledge is acquired through a process that is individual and 
personal. That is, tacit knowledge is acquired with little explicit support from sources 
external to the learner, such as conventional classroom curricula. Second, tacit knowledge 
contains information regarding the appropriate responses to particular situations. This 
knowledge could be associated with understanding in a particular situation the 
consequence of one’s own actions or those of other involved parties. This feature 
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illuminates the action or procedural orientation of tacit knowledge as opposed to simply 
fact or declarative orientation. Third, tacit knowledge is applicable to the accomplishment 
of personally valued goals. Goals that are unrelated to an individual’s success typically 
do not lead to the acquisition of an extensive body of tacit knowledge, as the attention 
required for learning is devoted elsewhere. 

Though tacit knowledge tends to be difficult to articulate verbally, it is 
demonstrated by an individual’s general capacity to solve problems of a practical nature 
(Sternberg et al., 2000). That is, tacit knowledge plays a key role in practical 
intelligence—the ability to adapt to, select, and shape environments in order to solve 
everyday problems. According to the theory, tacit knowledge is applied to experience 
through cognition at two-levels of abstraction. The first is a higher-order problem solving 
process, which engages certain executive functions called metacomponents that include: 
problem identification, resource allocation, representation and organization of 
information, formulation of solutions, monitoring of solutions, and evaluation of solution 
outcomes. The second is a lower-order set of cognitive processes called performance 
components, through which action is executed (e.g., analogic reasoning, induction, and 
inference.) 

Based on the recognition that tacit knowledge cannot be directly measured, one of 
the ways it has been measured has been through its application to the solution of practical 
problems. Described in detail in Sternberg et al. (2000), the process of generating and 
selecting problems to be used in the assessments is geared toward developing problems 
that tap knowledge having the characteristics described above. Studies conducted by 
Sternberg and his colleagues (see Sternberg et al., 2000 for a summary) indicate that 
results of assessments of tacit knowledge are related to the performance of military 
leaders and that results of such assessments explain leadership effectiveness above and 
beyond frequently used measures of general cognitive ability. Thus, by facilitating the 
acquisition of tacit knowledge, the metacognitive skills for practical problem solving may 
be more effectively used and performance—the demonstration of practical intelligence— 
consequently enhanced. 

Sternberg and his colleagues (Sternberg et al., 2000; Sternberg & Wagner, 1992; 
Wagner & Sternberg, 1991) further describe three cognitive processes that underlie the 
acquisition of tacit knowledge: selective encoding, selective combination, and selective 
comparison of information. These processes interact to reduce and organization the vast 
amount of information in any given situation. Selective encoding involves extracting 
information that is relevant to personally valued goals. Through selective combination 
this encoded information is integrated to form a meaningful pattern. This new pattern of 
information is compared relative to previously acquired tacit knowledge by the process of 
selective comparison. 

Tacit knowledge that is adaptive should enhance the metacognitive skills used in 
practical problem solving. Thus, by increasing the effectiveness of selective encoding, 
combination, and comparison of information, the acquisition of tacit knowledge may be 
facilitated and practical problem solving improved. Described in more detail later, 
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Sternberg, Okagaki, and Jackson (1990) and Sternberg, Wagner, and Okagaki (1993) 
have investigated with some success the effectiveness of training interventions targeting 
these cognitive processes. 

Through extensive qualitative research in the domain of military leadership, 
researchers found that tacit knowledge in military leadership could be divided into three 
broad categories: intrapersonal tacit knowledge (managing self), interpersonal tacit 
knowledge (managing others), and organizational tacit knowledge (solving organizational 
problems). Within each category, the relevance of a particular aspect of tacit knowledge 
was appropriately related to the rank of the military leader and reflected the shift in job 
demands across ranks (Donnithome, 1993; Horvath, et al,. 1996,1998,1999). For 
example, tacit knowledge about managing oneself (e.g., temper control, time 
management) has been reported by military personnel as particularly relevant to the job 
demands of military leadership for lieutenants, captains, or lieutenant colonels. In 
contrast, interpersonal tacit knowledge regarding the development of subordinates was 
shown to be primarily relevant to the jobs of captains in their verbal reports. The patterns 
of the relevance of tacit knowledge across ranks reflect the shift in job demands—from 
motivating subordinates using immediate, face-to-face leadership to fostering the future 
of the Army through personnel development and vision—which occurs as military 
leaders are promoted (Donnithome, 1993). They also may indicate the psychological 
development of military leaders as they progress upward in rank. 

In summary, it appears that one path to facilitating the acquisition of tacit 
knowledge in military leaders may be through the development of more effective 
cognitive processes that selectively encode, combine, and compare information present in 
the occupational environment (for more see Sternberg et al., 1990, 1993). Through the 
reduction and organization of information, the acquisition of tacit knowledge would 
influence the effectiveness and flexibility of the metacognitive skills used to solve 
practical problems in occupational environments, thus enhancing practical intelligence 
and development of expertise. 

Facilitating Tacit Knowledge 

Reflection on Condition 

By outlining the cognitive information processes underlying tacit knowledge and 
practical intelligence, the theory described above suggests the characteristics that efforts 
to develop tacit knowledge acquisition should share. That is, such efforts should involve 
the environmental context in which tacit knowledge is situated and yet should provide the 
individual with a somewhat more general facility to acquire tacit knowledge from 
experience. In the interests of developing a teaching strategy for improving the ability 
with which students adapt to, select, and shape their environments, Sternberg (1998) 
presented twelve principles for translating his theory of successful intelligence into 
specific instructional methods for schoolteachers. These principles provide an example of 
an instructional method geared toward facilitating tacit knowledge acquisition in a 
particular environment while simultaneously suggesting general methods for learning 
from experience. Although practical intelligence represents but a component of the theory 
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of successful intelligence, which also describes the critical contribution of analytical and 
creative intelligence to the adaptation to, selection of, and shaping of environments, we 
focus here on practical intelligence, as it appears to be especially relevant to military 
leadership. A subset of the twelve principles related to practical intelligence that inform 
the development of experimental reflection methods in this research is described below. 

Principle 1. The goal of instruction is the creation of expertise through a well- and 
flexibly organized, easily retrievable knowledge base. 

To accomplish this goal, instruction should facilitate the process through which 
tacit knowledge cognitive structures become less ambiguous and increasingly context- 
specific. As the unaided acquisition of tacit knowledge occurs through experience in 
actual occupational environments, the facilitation of tacit knowledge acquisition in an 
experimental setting should provide insight into how such acquisition occurs to maximize 
its effectiveness and relevance (Horvath et al., 1999). Experimental reflection methods 
that make tacit knowledge explicit and available for conscious evaluation and 
development should be transferable as a learning strategy that military leaders can use 
adaptively in the field. Therefore, ideally the experimental reflection methods should be 
techniques for enhancing tacit knowledge and practical intelligence in the field both 
during peacetime and in combat. 

Principle 2. Instruction and assessment should involve utilization, at various 
times, of all six metacomponents of the problem-solving cycle. 

Facilitation of tacit knowledge acquisition is stimulated by both the recognition of 
relevant information and associated actions that when applied to real-world situations 
result in effective problem solving. This process involves the previously specified six 
metacognitive skills (problem identification, resource allocation, the representation and 
organization of information, strategy formulation, monitoring of problem-solving 
strategies, and evaluation of problem solutions). Because tacit knowledge cannot be 
directly measured, these metacomponents can form the basis for assessing as well as 
developing tacit knowledge and practical problem solving. 

3. Instruction should involve utilization, at various times, of at least three 
knowledge-acquisition components, including (a) selective encoding, (b) selective 
comparison, and (c) selective combination. 

Findings described in Sternberg et al. (1993) suggest that targeting the cognitive 
processes underlying tacit knowledge acquisition (i.e., selective encoding, combination, 
and comparison) is critical for enhancing performance. In this research, groups of 
students who received training in knowledge-acquisition components were compared to 
groups of students who did not receive training. Trainees who received cues for selective 
encoding or selective comparison targeted more relevant information, missed fewer 
relevant pieces of information, and labeled fewer pieces of irrelevant information as 
relevant than did the control groups. 
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4. Instruction should help students (a) adapt to, (b) shape, and (cl select 
environments. 


As described previously, the primary function of tacit knowledge and practical 
intelligence is to aid the individual in adapting to, shaping, and selecting environments 
that are practical in nature (e.g., resolving a conflict at work, performing well on the job, 
etc.). With practice metacognitive skills can be developed and improve upon problem 
solving through conscious attention to problem identification, resource allocation, 
representation and organization of information, formulation of solutions, monitoring of 
solutions, and evaluation of solution outcomes. The ultimate goal of methods designed to 
development practical problem solving skills should be relevant to the long-term 
development of expertise—an adaptive and flexible overall capacity to effectively relate 
to the practical world (e.g., see Smith, Ford, & Kozlowski, 1997). Such a relationship 
would allow military leaders to better adapt to environments to which they are initially 
ill-suited, to select environments that allow them to capitalize on their strengths and 
compensate for their weaknesses, and to shape environments that would maximize both 
their learning and performance outcomes. 

Reflection on Action 

Past work shows the often-remarkable ability that individuals have to demonstrate 
effective knowledge without awareness (Schon, 1983). However, it also shows that 
tacitly held beliefs are frequently maladaptive (e.g., Argyris, 1988). Instead of attempting 
to characterize the nature and content of effective tacit knowledge, several investigators 
(Argyris, 1988; Epstein, 1999; McCall et al., 1988) have sought to investigate the effects 
of maladaptive tacit knowledge and the processes through which this knowledge is 
updated or changed. Agreeing that the ability to learn from experience is critical to job 
success, investigators taking this approach have devoted a great deal of research to 
developing models of on-the-job learning and to facilitating the acquisition of tacit 
knowledge in occupational environments (e.g., McCall et al., 1988; Myers & Davids, 
1993; Nonaka, 1994; Raelin, 1997; Schon, 1983; Smith et al., 1997). Like the work of 
Sternberg and his colleagues, the work of these researchers emphasizes the importance of 
developing a flexible, adaptive body of knowledge that can be applied to problem solving 
in occupational environments in which the problems are complex and ill defined. The 
findings from research and practice using this approach suggest that personal reflection 
guided by expert knowledge is critical for improving what is learned from experience—a 
practice used for instructing cadets in military leadership at West Point (McNally, Gerras, 
& Bullis, 1996). 

Broadly defined, reflective practice is the use of introspection, typically initiated 
by unexpected action outcomes, to determine the relationship between tacitly held 
theories regarding the meaning of conditions, the relevance of actions and the related 
outcomes of those actions (Aryris, 1999; Epstein, 1999; Mezirow, 1991; Raelin, 1997). 
Ideally, reflection is integrated with action so that “reflection-in-action” occurs when to 
modify knowledge structure and respond in ways that are more adaptive in a particular 
situation (Schon, 1983). However, Schon (1983) noted that the timeline defining action 
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might range in length from just a few seconds (e.g., kicking a field goal) to several 
months (e.g., preparing a legal case) to even an entire career. He also observed that the 
shorter action timelines are less likely to benefit from reflection than the longer ones, as 
there is more opportunity for reflection to interrupt the action process. Reflection, 
however, may also occur after an action has been taken. Indeed, several reflective 
methods described in the literature involve introspection on actions already taken rather 
than introspection during the course of action (Epstein, 1999; Raelin, 1997). Still, the 
goal of such methods is to facilitate learners’ ability to reflect in real time. 

The utility of reflective methods in organizations is apparent when one notes that 
articles or book chapters describing such methods are typically found in publications 
geared toward practitioners (Epstein, 1999; Osterman & Kottkamp, 1993; Raelin, 1997, 
Wegner & Snyder, 2000) and discuss the use on reflection in the field (e.g., Argyris, 
1999; McCall et al. ; 1988; McNally et al., 1996). However, the effects of reflection on 
occupational effectiveness are not fully known and there are diverse points of view 
regarding its usefulness with regard to identifying and changing behavior (e.g., Mezirow, 
1991; Nisbett & Wilson, 1977; Nonaka, 1994; Schon, 1983). Further, theory regarding 
the processes through which reflective practice has its influence is diverse. Nevertheless, 
there do appear to be some common threads that tie this body of work together. One 
common theme identified as the basis of effective reflection is a process of hypothesis 
testing (Mezirow, 1991). Many scholars agree that critical steps to updating knowledge 
and improving performance include developing ideas about the rationale for proceeding 
with a particular course of action, following up on those ideas by actually taking action, 
and then reflecting on the results (Argyris, 1999; Epstein, 1999; Schon, 1983). 

While various reflection methods have been proposed in the literature (e.g., 
Argyris, 1999; Epstein, 1999; McNally et al., 1996; Raelin, 1997), there is general 
agreement on how these methods facilitate learning from experience, which can be 
reflected in performance outcomes. These points of agreement served as a springboard 
from which three principles were derived for developing the reflection methods designed 
for this investigation. 

Principle 1. Reflection methods must direct attention to the precise condition- 
action relations that lead to the outcome of that action relative to some known 
standard. 


As noted previously, hypothesis testing—or testing of the tacitly held condition- 
action propositions—is critical to effective reflection and facilitating knowledge 
acquisition (Mezirow, 1991; Schon, 1983), as does the interpretation of the meaning of 
the conditions - a form of knowledge representation.. Reflection on action and its 
consequences facilitates learning by requiring the learner to assess the accuracies of his 
interpretations of the meanings of the condition, its appropriate categorization and 
elaboration relative to some known standard. The standard may reflect either personally 
or culturally valued behavioral goals. This alerts the learner to the possibility that there 
are tacit influences on decision-making (e.g., Argyris, 1988, 1999; Schon, 1983) and tacit 
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beliefs or assumptions regarding the condition-action relationships in the environment 
that can ultimately limit action outcomes (Schon, 1983; Tolman & Brunswik, 1935). 

Principle 2. Reflection methods must direct attention to the tacit condition-action 
and action-outcome propositions that gave rise to the action taken. 

Awareness of the fundamental link between tacitly held condition-action and 
action-outcome relationships is a critical step in improving decision-making and the 
primary goal of reflective practice (Argyris, 1988; Schon, 1983). Reflection increases 
awareness through a process of examining tacitly held beliefs associated with the 
problem situation in which an expected outcome did not follow a chosen course of action. 
By considering alternative assumptions that might account for the unexpected outcome, a 
new understanding of condition-action relationships in the situation is developed. Based 
on this new understanding, alternative courses of action can be considered and tested in 
practice to confirm its validity (Epstein, 1999; Schon, 1983). 

Principle 3. Initially, reflective practice should be guided by feedback, preferably 
that of an interested expert, so that newly developed condition-action 
propositions are less likely to be maladaptive. 

People may benefit differentially from reflective practice, depending on how 
adept they are at identifying the relevant condition-action contingencies (Nisbett & 
Bellows, 1977; Wilson & Nisbett, 1978). Expert feedback, which has been associated 
with the development of expertise (McNally et al., 1996), may be effective because it 
directs attention appropriately. This implies that methods that focus reflection on the 
appropriate condition-action contingencies have the potential to reduce individual 
differences associated with reflective practice. 

CONCEPTUAL FRAMEWORK 

As stated previously, the two approaches to understanding tacit knowledge 
presented above are complementary. Further, their corresponding implications for 
facilitating tacit knowledge acquisition, the principles derived from the theory, can be 
integrated into a comprehensive framework for understanding the role that both the 
condition and action aspects of tacit knowledge play in practically intelligent 
performance. Reproduced and slightly modified on the next page in Figure 1 is a 
theoretical framework presented in Antonakis, Hedlund, Pretz, and Sternberg (2001) that 
illustrates how these approaches fit together. Elements in the shaded portion of the figure 
represent the tenets of Sternberg’s theory regarding practical intelligence and the role that 
tacit knowledge plays in the demonstration of practical thinking. Non-shaded elements 
represent the tenets of Schon’s theory regarding the role of action and reflection in 
reorganizing tacit knowledge. The condition and action aspects of tacit knowledge and 
practical problem solving are represented in the top and bottom halves of the figure. 
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Figure 1. Model of practical intelligence and tacit knowledge acquisition 


Illustrating Sternberg’s work, the framework draws clear links between the 
cognitive processes underlying knowledge acquisition and the role that knowledge 
acquisition plays in supporting the cognitive metacomponents engaged in practical 
problem solving. Performance components are involved in the execution of decisions 
made by an individual. Activation of the performance components leads to a particular 
action, which leads to an outcome. 

Illustrating Schon’s work, the action taken provides a test, the outcome of which 
will either serve to support existing condition-action propositions held by the individual 
or to challenge them. The connection between reflection on action outcomes and tacit 
knowledge is shown to occur in two possible ways. First, shown in the middle of the 
figure, when expected action outcomes occur, the tacitly held condition-action 
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propositions are supported. If action outcomes do not occur as expected, this connection 
represents the adjustment of action or application of tacit knowledge. 

The second connection between reflection and action outcomes, shown on the 
lower left-hand side of the figure, indicates the path of re-integration of knowledge and 
reformulation of condition-action propositions that occurs when unexpected outcomes 
occur as a result of action. This connection occurs when the individual must reframe the 
problem. Together, the top and bottom halves of the figure clearly indicate the potential 
benefit of focusing on the condition aspects of tacit knowledge, the action aspects of tacit 
knowledge, or both, when attempting to facilitate tacit knowledge acquisition. 

Based on this model, reflection methods utilized in this investigation target the 
condition aspect, action aspect, and a combined approach. The differential effects of 
these methods were examined. Further, the combined approach was explored relative to 
traditional methods of analytical reflection typically employed in academic settings. 

TACIT KNOWLEDGE MEASURES AND REFLECTION METHODS 

Tacit Knowledge Measures 

In general, two types of situational-judgment testing formats have been developed 
to assess practical problem solving and domain-specific tacit knowledge. Tacit 
knowledge inventories feature a series of brief vignettes, each of which present a domain- 
specific problem situation and provides a set of solution alternatives, which are rated for 
quality on a Likert scale. Case study scenarios present more extended problem situations 
that simulate the contextual complexity of real-world situations by including multiple 
issues, previous actions taken, and some relevant information needed to understand and 
solve a complex problem. They consist of a detailed description of the particulars of a 
problem situation as it unfolds over time, followed by a set of open-ended questions 
designed to assess knowledge-acquisition components and practical problem-solving 
skills, two important aspects of behaving in a practically intelligent way (Sternberg, 

1988). This method of assessing tacit knowledge draws from approaches commonly used 
in managerial assessment and education, including in-basket tests, which require 
prioritizing and responding to job-relevant materials (e.g., memos and reports) in a 
limited amount of time, and case studies, which involve critiquing and/or solving a 
detailed case description. Tacit knowledge vignettes and case study scenarios are derived 
from subject matter experts and are designed to represent realistic practical problems that 
might be encountered in a particular domain. 

Two new platoon-level military scenarios and two new college life scenarios were 
developed for this investigation to measure practical problem-solving skills and domain- 
specific tacit knowledge. Procedures employed to develop and score each type of 
scenario are described below. 
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Military Scenarios 

To develop military scenarios, interview data and corresponding coded tacit 
knowledge analyses previously obtained from interviews with Army officers to develop 
the Platoon-Level Tacit Knowledge for Military Leaders (TKML) Inventory (Hedlund et 
al., 1999) were reanalyzed to create two new tacit knowledge scenarios. First, two of the 
problem themes inherent in these data (e.g., difficulties with a Sergeant First Class) were 
selected to provide a contextual framework for the new scenarios. Next, specific tacit 
knowledge condition-action statements relevant to each theme were incorporated into 
scenario content (e.g., when a Soldier is promoted without an increase in rank, it can be 
difficult to establish authority). To enrich scenario context, pertinent information was 
extracted from Army leadership doctrine, training materials, and standard operating 
procedures (Frame & Lussier, 1999; U.S. Department of the Army, 1994a, 1994b, 1998a, 
1998b, 1999a, 1999b, 1999c, 1999d, 2001,2002). 

Completed scenario drafts were reviewed by two senior Army officers who 
served as subject matter experts (SMEs), and were revised accordingly. The revised 
scenarios were piloted with a group of 20 senior non-commissioned and three 
commissioned officers in the 1 st Battalion, 1 st Infantry Regiment at West Point. Open- 
ended questions designed to measure practical problem solving and tacit knowledge were 
included in the scenarios. These questions were developed consistent with the principles 
of assessment and measurement such that each question 1) was a single question, rather 
than compound question, 2) was worded using Army terminology, rather than technical 
psychological terminology, and 3) was designed to be easy to read, rather than using 
complex phrasing. The new military tacit knowledge scenarios with accompanying 
questions are presented in the Appendix A. 

To assess the practical problem-solving skills and tacit knowledge content in the 
military scenarios, two comprehensive analytic scoring rubrics were developed to score 
responses to questions in each new scenario. An analytic approach to scoring emphasizes 
a more quantitative, point-by-point rating scale that focuses on the presence or absence of 
specific elements in the participant’s response (Mertler, C.A., 2001). In this case scores 
were assigned based on the extent to which responses demonstrated appropriate tacit 
knowledge content and relevant metacognitive processes (i.e., condition-action-outcome 
linkages). Regarding tacit knowledge content, responses were evaluated by comparisons 
with relevant tacit knowledge content previously obtained through research from Army 
experts. Regarding metacognitive processing, responses were evaluated in terms of 
properties associated with problem identification, selected course of action, anticipated 
action outcomes, and obstacles. For example, problem identification was scored based on 
evaluation of immediate vs. long-term time perspective, relative importance of problem 
identified, and complexity of the knowledge network reflected by cause and effect 
linkages identified. The selected course of action was scored based on evaluation of the 
complexity of the procedural knowledge reflected by causal linkages to identified 
problems and strategies for action. Anticipated action outcomes and obstacles were 
scored based on evaluation of the specificity of condition-action linkages and level of 
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abstraction and insight. Scoring rubrics for new military scenarios are displayed in the 
Appendix B. 

College Life Scenarios 

To develop college life scenarios, interview data were collected from 
upperclassmen serving as residence hall advisors, who were considered “subject matter 
experts” in college life because of their experience and demonstrated ability to succeed in 
college (details of the procedures can be found in the Appendix C). Interviewees were 
asked to talk about a problem faced in college in which they learned an important lesson 
about how to succeed in some aspect of college life. They were encouraged to describe 
the problem situation in detail, how they handled it, and what they would have done 
differently in hindsight. 

College life scenarios were developed based on common themes and dilemmas 
that emerged from the interviews with college life experts. Two context-rich, complex, 
and ambiguous scenarios were created to elicit variability in problem identification and 
solution generation. The first scenario features multiple problems associated with 
performance in an upper-level English class. The second scenario depicts a situation in 
which a roommate exhibits a disruptive pattern of behavior and declining performance in 
school. Open-ended questions designed to measure practical problem solving and tacit 
knowledge were incorporated into the scenario. Scenario drafts were piloted with a group 
of upperclassmen who considered the scenario independently, responded to open-ended 
questions, and then participated in a focus group discussion to assess content and 
questions. Scenarios were revised accordingly. College life scenarios with accompanying 
questions are displayed in the Appendix D. 

To assess the practical problem-solving skills and tacit knowledge content in the 
college life scenarios, two comprehensive scoring rubrics were developed that integrated 
aspects of analytic and holistic approaches to scoring. This revised approach was 
undertaken because it improved time efficiency without detracting from quality. 
Responses to scenario questions were rated holistically-on a five-point scale by recent 
college graduates (i.e., experts) based on two dimensions: 1) tacit knowledge content, and 
2) complexity of thought or understanding of cause and effect. Scoring rubrics for new 
college life scenarios are displayed in the Appendix E. 

Reflection Methods 

For the military research effort, three reflection methods were developed to test 
the relative effect of reflection that focus on different aspects of practical problem solving 
based on the previous discussion. Each method was designed to examine thinking 
associated with practical problem solving in response to one of the tacit knowledge 
vignettes, a case study scenario, and a practical problem that the respondent had dealt 
with in their own life. Each intervention began with a description of the particular 
strategy for reflection. The condition-focused method facilitated reflection on problem 
identification and goal formulation associated with a tacit knowledge vignette. It included 
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questions that uncovered factors considered and probed underlying assumptions 
associated with problem definition and solution goals. For example, “Describe the factors 
you considered when choosing your goal and determining the problem that must be 
solved (e.g., doctrine, personal values, assumptions about Army culture, procedures, and 
personnel, knowledge based on previous experiences).” The action-focused method 
facilitated reflection on the link between action and action outcomes associated with a 
particular vignette. It included a guided comparison of the subjects own response rating 
to the average response of an expert group with questions that probed alternative 
assumptions about goals and action outcomes. For example, “Perhaps the ( comparison 
expert group ) and I had similar goals and identified a similar problem, but we preferred 
different actions. What outcome do you feel your actions would achieve? What outcome 
do you feel the students action would achieve?” The condition and action method 
facilitated reflection on aspects of both condition and action by incorporating elements of 
both the condition and action reflection methods. It included a guided comparison of 
subjects’ own responses to average expert responses in terms of problem identification, 
goals, actions, and anticipated action outcomes. Sample reflection materials are displayed 
in Appendix F (1 - 3). 

For the college life research effort, a reflection control condition was designed to 
provide a more stringent differential test of reflection methodology. It included two 
articles on relevant practical issues in college life, followed by a series of open-ended 
analytic questions pertaining to these articles. Questions facilitated reflection on personal 
reactions to the issues presented and examination of the relative benefits and drawbacks 
of solutions to issues presented. For example, a sample question on the topic of co-ed 
living is “What advantages and disadvantages do you see to having co-eds share 
bathrooms?” A sample reflection control is displayed in Appendix F (4). 

EMPIRICAL STUDIES 

Overview 

Two related experimental studies were conducted to investigate the effect of 
reflection methods on practical problem solving in Army officer and college student 
samples using domain-specific and context-dependent measures that were developed to 
assess tacit knowledge in each sample. The first research effort involving an Army officer 
sample was discontinued midway through data collection because access to officers was 
suspended due to the Iraq War. The second research effort was designed to complete the 
testing of hypotheses with a college student sample. 

Research Effort One: Developing Tacit Knowledge in Military Leadership 

Based on the aforementioned conceptual discussion, it was predicted that training 
in reflection methods that focus on condition, action, or a combination of condition and 
action would improve practical problem solving as measured by performance on tacit 
knowledge assessments more than no such training. It was also predicted that reflection 
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methods that combine reflection on condition and action would be superior to those that 
focus on either condition or action independently. 


Hypotheses 

HI: Training using reflection methods will improve practical problem solving 
more than no reflection control. 

H2: Training using methods that focus on both reflection on condition and 
reflection on action will improve practical problem solving more than either 
approach alone. 

Method 


Subjects 

One hundred one (101) Army officers representing various branches at three 
Army bases in the Northeast and Midwest participated in the research effort. Twenty-nine 
(28.7%) were lieutenants (LTs), 30 (29.7%) were captains (CPTs), 22 (21.8%) were 
majors (MAJs), and 20 (19.8%) were lieutenant colonels (LTCs). Ninety-seven (96%) 
were men and 4 (4%) were women. Participants ranged in age between 22 and 50 years. 
The mean age of the participants was 32.9 years with a standard deviation of 6.8 years. 
One participant did not report age. Ninety-nine (98%) participants were native English 
speakers; one participant was not, and one did not report language status. Forty-eight 
(47.5%) reported non-military leadership experience (e.g., restaurant manager, youth 
leader, sports captain, etc.), and 51 (51.5%) did not. Years in service ranged from 0 to 23 
years; the mean was 10.6, with a standard deviation of 6.8 years. One participant did not 
report the number of years in service. The number of years in current rank ranged from 0 
to 6 years with a mean of 2.2 and standard deviation of 1.5 years. Five participants did 
not list the number of years spent in their current rank. Table 1 presents the distribution 
of rank in each of the three experimental and control conditions. As can be seen in the 
table, the distribution of rank across conditions was largely consistent. However, due to 
limitations associated with sample access, the distribution of rank within each condition 
was variable. 

Table 1. Officer rank across conditions. 



Experimental Condition 

Total 

Control 

Condition 

Combined 

Action 

Rankl 11t or 21t 

2 

10 

9 

8 

29 

cpt 

14 

6 

9 

1 

30 

maj 

5 

1 

12 

4 

22 

Itc 

5 

11 

0 

4 

20 

Total 

26 

28 

30 

17 

101 
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Procedure 


Army officers participated in a single three-hour session. Participants were told 
the purpose of the research was to test methods to improve acquisition of experience- 
based knowledge in military leadership. Rank appropriate materials were distributed to 
participants. Demographic and motivation covariate surveys and a self-paced tacit 
knowledge pre-test were administered to participants followed by a short break (10-15 
minutes). Following the break, participants received one of three experimental 
interventions (30-40 minute self-paced reflection exercise) or were placed in the control 
condition, in which they completed another tacit knowledge scenario. The control 
condition scenario contained similar instructions and was similar in structure to pre-and 
post-test scenario measures. Following another short break, a self-paced tacit knowledge 
post-test, a Satisfaction with Intervention survey (experimental conditions only), and 
cognitive ability tests were administered. 

Measures 

Tacit Knowledge and Practical Intelligence. Tacit knowledge and practical intelligence 
were assessed with tacit knowledge inventories and extended case study scenarios. 
Measures were counterbalanced to control for potential differences in difficulty. 

However, because access to military officers was discontinued during the course of the 
research, administration of measures according to the counterbalancing design was 
incomplete. 

Tacit Knowledge in Military Leadership (TKML; Horvath, Hedlund , Snook, Forsythe, & 
Sternberg, 1998; Hedlund, Williams, Horvath, Forsythe, Snook, Wattendorf, McNally, 
Sweeney, Bullis, Dennis, & Sternberg, 1999). This survey features brief problem 
situations that may be encountered in Army leadership. There are three versions: one 
each for platoon-level, company-level, and battalion-level leadership, each of which 
contains 18 items. Eight vignettes were selected from each version of the TKML based 
on higher levels of agreement among experts on response options (i.e., the 70% 
confidence intervals around the expert mean ratings were smallest). A block of four 
vignettes from each tacit knowledge inventory was used as a pre-test, and a block of the 
other four was used as a post-test. Participants received rank-appropriate TKML vignettes 
(i.e., LTs received platoon-level TKML vignettes, CPTs received company-level 
vignettes, and MAJs/LTCs received battalion-level vignettes). Internal consistency 
estimates of the TKML ranging from 0.66 to 0.76 and evidence of construct and 
criterion-related validity have been reported. 

Extended Case Study Scenarios. Two platoon-level case study scenarios that were 
developed for this research feature leadership problems that might be encountered by a 
platoon leader. The first scenario involves a unit that has undergone multiple changes in 
leadership and subsequent problems with discipline, performance, and chain of 
command. The second scenario involves problems associated with a first sergeant who 
exhibits a pattern of poor performance, possibly related to alcohol abuse. The scenarios 
include six open-ended questions designed to assess practical problem solving and tacit 
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knowledge in military leadership. A sample question is “What problems need to be 
addressed in this situation?” Participants responded to questions by speaking into digital 
voice recorders. Three trained research assistants scored transcribed responses. Inter-rater 
reliabilities, estimated by an intraclass correlation (two-way mixed effects), were .85 for 
PS2 and .84 for PS3. 

Cattell Culture Fair Test of g (Cattell & Cattell, 1973). This test is designed to 
assess intelligence equivalently across cultural groups using non-verbal stimuli. The full 
test contains three scales, each of which is made up of multiple subtests. Two, timed 
subtests that require participants to select patterns from among figures from scale 2 were 
administered to measure fluid cognitive ability. The first subtest has 10 items and the 
second, 14 items. Full-scale reliabilities have been reported ranging from .85 to .91. 

Mill Hill Vocabulary Scale (Raven, Raven, & Court, 1985). This un-timed, 66- 
item test of vocabulary is designed to measure crystallized cognitive ability. Participants 
are presented with a word and must select from among four options the closest synonym 
to the word presented. A short-form of the test containing 33 items was administered in 
this research. Full test split-half reliability of .90 and test-retest reliabilities of .87 to .95 
have been reported. 

Attitudes Towards Leadership Instruction. This survey was developed for this 
research to control for the effect of individual differences in the motivation to participate 
in leadership instruction. It contains eight evaluative statements about previous leadership 
instruction. A sample statement is, “I have found the leadership courses I have taken to 
be effective in helping me deal with leadership problems outside of the classroom.” 
Respondents indicate the extent to which they agree with the statement on a three-point 
scale (1 = disagree, 2 = neither agree or disagree, and 3 = agree). 

Satisfaction with the Intervention. This survey was developed for this research to 
control for the effect of possible differences in satisfaction with the interventions on post¬ 
test performance. Participants are asked to rate the degree to which they agree with a 
series of statements about the experimental intervention on a three-point scale (1 = 
disagree, 2 = neither agree or disagree, and 3 = agree). A sample item is “ I learned 
something from the intervention.” 

Results 


Descriptive Statistics 

Four participants completed rank-inappropriate versions of the assessments and 
one outlier was removed from the dataset. One significant outlier was also removed. 
Seven participants did not complete scenario measures and 10 failed to complete 
cognitive ability tests. Descriptive statistics are displayed in Table 2. 
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Table 2. Descriptive statistics 



N 

Mean 

SD 

1. TKML pre-test* 

96 

3.236 

1.322 

2. TKML post-test* 

96 

2.767 

1.290 

3. PS2 Seen, pre-test 

42 

5.425 

1.483 

4. PS2 Seen, post-test 

48 

5.394 

1.319 

5. PS3 Seen, pre-test 

48 

5.395 

1.319 

6. PS3 Seen, post-test 

42 

6.046 

1.513 

7. Cattell 

86 

12.907 

3.172 

8. Mill Hill 

86 

19.034 

3.411 


*Scores represent the Euclidean distance from the expert mean 
Reliability 

Measures of internal consistency for all assessments were acceptable when 
Spearman Brown correction was applied to correct for test length. Reliabilities are 
displayed in Table 3. 

Table 3. Reliabilities for measures in Research Effort 1 


Reliabilities for measures in Research Ef 

fort 1 

Measure 


N 

miHm 


8 



7 



8 

PS2 


5 

PS3 

HH 

5 

Attitude* 

§§9H^H| 

5 

Satisfaction* 


5 

Mill Hill 

0.91 

33 


SDlit-Half 


Cattell 

0.86 

24 


• item(s) with poor intercorrelations omitted 
Group Comparisons 

There were no significant mean differences between experimental and control 
groups on satisfaction, attitude, or cognitive ability covariate measures. There was no 
effect of version on TKML pre- or post-tests but evidence of a strong interaction of test 
version on scenario pre- and post-test scores (F (1,87) = 24.567, p = .000). Participants 
who received the PS2 in pre-test improved on the PS3 post-test. Participants who 
received PS3 in the pre-test declined slightly in the PS2 post-test, suggesting that the 
scenarios are not equivalent. Because of this finding and the incomplete administration of 
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the counterbalanced design, PS2 and PS3 scenario scores were not collapsed into 
combined pre-and post-test scores. Instead they were analyzed separately. 

Scenario Validity 

Scenarios, as compared to vignettes, require a different level of cognitive 
processing and mode of solution construction. They are information-rich relative to 
vignettes and require respondents to construct and articulate tacit knowledge content. In 
comparison, vignettes are relatively brief and respondents evaluate, rather than generate, 
response strategies. Moreover, regardless of format, tacit knowledge measures sample 
from a wide range of relevant, domain-specific knowledge. Therefore, individuals tend to 
have variable levels of experience pertaining to specific item content, which may 
decrease the likelihood that different measures will be highly related. While it is expected 
that tacit knowledge vignettes and scenarios should correlate, the nature of the construct 
and the differences in testing format would suggest modest correlations. 

Because PS2 and PS3 scenarios were not equivalent as noted previously, each 
scenario was analyzed separately. Construct validity was assessed by examining 
correlations between scores for each scenario with scores on the TKML tacit knowledge 
vignettes (see Table 4). As can be seen in the table, PS3 was significantly related to the 
TKML. TKML pre-test (distance squared scores) and PS3 scenario scores were 
negatively correlated (r (89) = -.25, p = .02). TKML post-test was also negatively 
correlated with PS3 (r (89) = -.24, p = .024). Correlations between the PS2 scenario and 
TKML pre- and post-tests were lower and did not reach significance (TKML pre-test: r 
(89) = -.132, p = .22; TKML post test: r (89) = -.136, p = .2). Correlations between 
scenario scores and cognitive ability measures were small to moderate. PS3, but not PS2, 
marginally correlated with the Cattell, r (80) = .204, p = .07. PS2, but not PS3, correlated 
with the Mill Hill, r (80) = .268, p = .016. The differential correlations between PS2 and 
PS3 scenarios and tacit knowledge vignettes (TKML), as well as cognitive ability 
measures (Cattell and Mill Hill) may relate to differences in scenario difficulty. 
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Table 4. Correlations among TK measures and cognitive ability covariates. 


Correlations 


1_ 

TKML pre 

TKML post 

Ps2 

Ps3 

Cattel 

Mill 

TKML** 

_ . , Pearson 

Pretest - 

1 

.358** 

-.132 

-.250’ 

-.249* 

-.071 


Sig. (2- 


.000 

.216 

.018 

.021 

.517 


N 

96 

96 

89 

89 

86 

86 

1 TKML***posttest Pearson 

.358** 

1 

-.136 

-.240* 

-.194 

-.110 


Sig. (2- 

.000 


.203 

.024 

.074 

.314 


N 

96 

96 

89 

89 

86 

86 

Ps2 

Pearson 

-.132 

-.136 

1 


.068 

.268* 


Sig. (2- 

.216 

.203 



.547 

.016 


N 

89 

89 

89 

89 

80 

80 

Ps3 

Pearson 

-.250* 

-.240* 


1 


.064 


Sig. (2- 

.018 

.024 




.572 


N 

89 

89 

89 

89 

80 

80 

Cattel 

Pearson 

-.249* 

-.194 

.068 


1 



Sig. (2- 

.021 

.074 

.547 



nSaSlia 


N 

86 

86 

80 

80 

86 

86 

Mill 

Pearson 

-.071 

-.110 

.268* 

.064 


1 


Sig. (2- 

.517 

.314 

.016 

.572 




N 

86 

86 

80 

80 

86 

86 


**. Correlation Is significant at the 0.01 level (2- 
*. Correlation is significant at the 0.05 level (2- 


***Distance scores should be interpreted in reverse 


Hypothesis Tests 

Tacit Knowledge Vignettes 

To test the differential effects of experimental reflection methods and no 
reflection (control group) on tacit knowledge vignette performance, a GLM univariate 
analysis of variance was conducted with the TKML post-test as the dependent variable, 
experimental condition as the independent variable, TKML pre-test as a covariate, and 
simple contrasts between experimental and control groups. Results show a strong effect 
of condition, F (3, 91) = 3.743, p = .014. Contrasts show that only the combined 
condition and action reflection method was significantly different from the control group, 
contrast estimate = -.696, p = .034. Comparisons show that participants in the combined 
group (“comb”) (M = 2.563, SE = .243) score better than those in the reflection on action 
group (“action”) (M = 2.641, SE = .323), and in the reflection on condition group 
(“cond”) (M = 3.209, SE = .243). Figure 2 displays the tacit knowledge marginal means 
in pre- (1) and post-test (2) for all experimental conditions. 

These results partially support Hypothesis 1 and fully support Hypothesis 2. Only 
the combined reflection method, but not the methods that focused on either approach 
alone, improved practical problem solving as measured by the TKML more than the no¬ 
reflection control condition. 
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experimental condition 

-control 

-cond 

--comb 

-action 


Figure 2. TKML marginal means pre (1) and post (2) test by experimental condition. 
Tacit Knowledge Scenarios 

As discussed previously pre- and post-test scenario scores were not compared 
because the PS2 and PS3 do not appear to be equivalent. To test the differential effect of 
experimental reflection methods and no reflection (control), the PS2 scenario was 
analyzed because there were an adequate number of participants who took it in each 
condition. The TKML pre-test score was used in place of a scenario pre-test score as a 
conceptually relevant proxy. A GLM univariate analysis of variance was conducted with 
the PS2 scenario post-test as the dependent variable, experimental condition as the 
independent variable, tacit knowledge vignette (TKML) pre-test and Mill Hill as 
covariates, with simple contrasts between the experimental and control groups. The Mill 
Hill covariate was included to provide an additional control for individual differences 
associated with verbal ability given the response format of the scenario differed from the 
vignettes. 

Results show a main effect of condition on post-test scores F (3, 41) = 4.414, p = 
.009. Simple contrasts between experimental and control groups show that reflection on 
action was significant (contrast estimate = 1.725, p = .002), reflection on condition was 
marginally significant (contrast estimate = .939, p = .081), but, unlike results using the 
tacit knowledge vignette, the combined reflection on condition and action method was 
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not significant (see figure 3). Because the scenario task was more complex than vignettes 
and the training and post-training integration time brief, participants may have needed 
more practice and integration time to benefit from the combined approach. 

These results provide partial support for Hypothesis 1 in that training methods 
that focused on either reflection on condition or reflection on action improved practical 
problem solving as measured by the PS2 scenario as compared to the no-reflection 
control condition. However, results did not support Hypothesis 2; the combined reflection 
on condition and action method was not superior to either approach alone in PS2 scenario 
post-test performance. 



experimental condition 

-control 

-cond 

comb 
-action 


Figure 3. PS2 scenario marginal means pre- (1) and post- (2) test by experimental 
condition. 







Discussion 


Taken together these findings provide preliminary evidence that tacit knowledge 
can be facilitated by reflection methods that are derived from the theory of practical 
intelligence. Findings were mixed with regard to the relative effectiveness of reflection 
methods, depending upon the type of tacit knowledge measure employed. The combined 
reflection method was effective in improving tacit knowledge scores in vignettes, which 
provide limited information about a specific problem situation and require assessment of 
specified response strategies. Reflection on condition and reflection on action methods, 
but not the combined method, were effective in improving tacit knowledge scores in the 
PS2 scenario, which may be due to the more complex nature of the scenario task. All of 
the information that must be managed in the scenario task may make it difficult for 
participants to benefit from the more inclusive method in a brief intervention. 

Research Effort Two: Developing Tacit Knowledge in College Life 

Consistent with previous theoretical and empirical work on the acquisition of tacit 
knowledge, it was predicted that the method that combines reflection on condition and 
action would improve practical problem solving as measured by performance on tacit 
knowledge assessments more than a traditional, analytic reflection method. 

Hypotheses 

H3: Training using methods that focus on both reflection on condition and action 
will improve practical problem solving more than analytic reflection. 

Method 


Subjects 

Participants were 235 college students from three universities in the Northeast and 
one in the Northwest. One hundred fifty-nine (68%) of the students were freshmen, 69 
(39%) were sophomores, 7 (3%) did not report their status. Participant age ranged from 
18 to 28 years with a mean age of 18.7 years. One hundred fifty-seven (67%) participants 
were women, and 78 (33%) were men. One hundred sixty eight (72%) were European 
American, 15 (6%) were Asian American, 11 (5%) were Hispanic American, 7 were 
African American (3%), 14 (6%) reported themselves as “other,” and 20 (8%) did not 
report ethnicity. Two hundred twenty-two (94%) reported being native English speakers, 
12 (5%) were not, and one participant did not report his native language status. 

Procedure 

Participants were paid volunteers ($30) who took part in a single three-hour 
session. They were randomly assigned to the experimental or analytic control conditions 
and told that the purpose of the research was to assess reflection methods for developing 
practical problem solving using materials that focused on college life. Covariate 
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surveys/tests and self-paced tacit knowledge pre-test materials were administered 
followed by a short break (10-15 minutes). Participants were instructed not to speak 
about research materials during breaks. After the break, participants completed either the 
experimental or control 30-40-minute self-paced reflection exercise. Following the 
intervention, participants took a more extended 25-minute break designed to mitigate 
fatigue and provide time for cognitive integration. During this time, food and drink were 
provided and participants watched a short segment of a comedy video. Following the 
break, they were encouraged to put forth the same level of effort as in the pre-test and 
tacit knowledge post-tests were administered. After completing the post-test, all 
participants completed the Satisfaction with Intervention survey. 

Measures 

Tacit Knowledge and Practical Intelligence. Tacit knowledge and practical intelligence 
were assessed using tacit knowledge inventories and extended case study scenarios. 
Measures were counterbalanced to control for potential differences in difficulty. 

College Student Questionnaire (CSQ: Sternberg et al., 2000). The CSQ is made up of 
vignettes that describe problem situations that might be encountered by a college student 
followed by possible solutions, the quality of which participants rate on a seven-point 
scale (1 = high quality, 7 = low quality). Two sets of six vignettes were selected based on 
content to form roughly equivalent pre- and post-test blocks. Adequate reliability and 
validity of these measures has been reported in Cianciolo et al., (2004). A general group 
consensus approach to scoring was applied, in which scores were the Euclidean distance 
from the mean squared. 

Extended Case Study Scenarios. Two college life case study scenarios, “English Class” 
and “Roommate,” were developed for this research and are described earlier in this 
report. Four independent trained raters scored English Class responses and three 
independent trained raters scored the Roommate responses. All of the raters were 
graduate students and recent college graduates. Scenarios were scored on two 
dimensions, response content and thoughtfulness. Scoring content involved comparison 
to expert responses and scoring thoughtfulness involved assessing the quality of 
procedural cause-and-effect linkages in responses. The total score is a mean of content 
and thoughtfulness scores. For the English Class scenario, average inter-rater correlation 
was 0.52 and coefficient alpha across four raters was 0.82. For the Roommate scenario, 
the average inter-rater correlation was also 0.52 and the coefficient alpha across three 
raters was 0.8. 

Cattell Culture Fair Test of g (Cattell & Cattell, 1961). This test of fluid cognitive ability 
was described in Research Effort 1. 

Mill Hill Vocabulary Scale (Raven, Raven, & Court, 1985). This test of crystallized 
cognitive ability was described in Research Effort 1. 
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Attitude Towards Instruction. This survey was developed to control for the effect of 
individual differences in the motivation to participate in reflection exercises. It contains 
six evaluative statements, for example, “I would be interested in attending a workshop on 
how to succeed in college life.” Respondents indicate the extent to which they agree with 
the statement on a five-point scale (1 = strongly disagree, 5 = strongly agree). 

Satisfaction with Intervention Survey. This survey was developed for this research to 
control for differential satisfaction with the reflection method on performance. It contains 
six evaluative statements, for example, “I learned something from the intervention.” 
Respondents indicate the extent to which they agree with the statement on a five-point 
scale (1 = strongly disagree, 5 = strongly agree). 

Results 


Descriptive Statistics 

Two cases were removed from the dataset because one subject did not complete 
post-test and another had consistently inappropriate answers suggestive of a motivational 
problem. Descriptive statistics are displayed in Table 5. A comparison of pre-test means 
suggests that the English Class scenario may have been somewhat easier than the 
Roommate scenario. 

Table 5. Descriptive statistics. Research Effort 2. 


Descriptive Statistics 



N 

Minimum 

Maximum 

Mean 

Std. 

Skewness 

Kurtosis 1 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Std. Error 

Statistic 

Std. Error | 

CLQ pretest (mean) 

235 

.77 

4.53 

2.3903 

.81470 

.547 

.159 

-.273 


CLQ posttest (mean) 

235 

.77 

5.29 

2.2775 

.87041 

.859 

.159 

.730 


Scenario pretest 

235 

9.50 

35.50 

25.4234 

4.20381 

-.343 

.159 

.551 

.316 

Scenario posttest 

235 

14.75 

36.25 

24.2046 

3.64294 

.045 

.159 

.531 

.316 

RM content pretest 

117 

1.54 

4.00 

2.8608 

.46111 

-.282 

.224 

.360 

.444 

RM thought pretest 

117 

1.25 

4.33 

2.9922 

.64382 

-.327 

.224 

-.096 

.444 

EC content pretest 

118 

1.17 

4.33 

3.0862 

.54091 

-.202 

.223 

.481 

.442 

EC thought pretest 

118 

1.00 

5.00 

3.1610 

.78278 

-.150 

.223 

.347 

.442 

RM content posttest 

118 

1.71 

4.00 ’ 

2.7560 

.38833 

.172 

.223 

.787 

.442 

RM thought posttest 

118 

1.33 

4.17 

2.7578 

.59766 

.142 

.223 

-.331 

.442 

EC content posttest 

117 

1.96 

4.08 

2.9734 

.48470 

.123 

.224 

-.430 

.444 

EC thought posttest 

117 

1.00 

5.00 

3.0385 

.85640 

-.364 

.224 

.134 

.444 

EC pre 

118 

1.19 

4.34 

3.0890 

.49897 

-.426 

.223 

1.258 

.442 

RMpre 

117 

1.50 

3.94 

2.9046 

.47845 

-.379 

.224 

.143 

.444 

Cattell total 

235 

5 

19 

12.49 

2.782 

-.164 

.159 

-.332 

.316 

Mill Hill total 

235 

4 

28 

16.27 

3.256 

.207 

.159 

.888 

.316 

Attitude total (sum) 

234 

12 

27 

19.60 

3.043 

-.149 

.159 

-.459 

.317 

Satisfaction total (sum) 

234 

9 

29 

21.93 

3.382 

-.828 

.159 

1.504 

.317 

Valid N (listwise) 

0 










Reliability 

Measures of internal consistency for all measures were acceptable, except for the 
Attitude toward Instruction Survey, which did not have a sufficiently high level of mean 
inter-item correlation to be used in final analyses. Reliability measures for the Mill Hill 
and Cattell cognitive ability tests were low. When a correction for test length was they 


27 


























reached marginally adequate levels. Reliabilities of cognitive ability tests may have been 
lower in this sample because it was more likely to evoke test anxiety than in the Army 
officer sample. College students are regularly tested on coursework content, but not 
cognitive ability. In contrast, Army officers are accustomed to regular tests of cognitive 
ability. Alternatively the college students may have been less motivated. Cognitive ability 
test means were also lower in the college student as compared to the Army samples, 
which would not be otherwise expected. Results are displayed in Table 6. 

Table 6. Internal Consistency estimates for Research Effort 2 measures. 



Alpha 

n 

College Student Questionnaire 

0.78 

12 

English Class Scenario 

0.74 

8 

Roommate Scenario 

0.71 

9 

Attitude toward Instruction 

0.45 

6 

Satisfaction with Intervention 

0.72 

6 

Mill Hill 

0.60 (.75) 

33 


SDlit-Half 


Cattell 

0.42 (.59) 

24 


Scenario Validity 

The construct validity of scenario measures was assessed by correlations with 
CSQ tacit knowledge vignettes, which are displayed in Table 7. For reasons discussed in 
the description of Research Effort 1, modest correlations between tacit knowledge 
scenarios and vignettes were expected. 

Scenario pre-test scores, which combine English Class and Roommate scenario 
pre-tests, correlate significantly with the CSQ vignette pre-test (r (233)= -.162, p = .013), 
which suggests convergent validity. As might be expected given measurement method, 
the CSQ pre-test correlates with the Catted* (r (233) = -.198, p = .002), but not the Mill 
Hill, and the scenario pre-test correlates significantly with the Mill Hill, (r (233) = .263, p 
= .000) and approach significance with the Cattell (r (233) = .108, p = .1). 

Group Comparisons 

There were no significant mean differences between the experimental and control 
groups on gender, ethnicity, the Satisfaction with Intervention survey, the Mill Hill, or 
the Cattell. There was no main effect of test version on the CLQ pre- and post-tests. 
However, there was evidence of a strong interaction of test version and pre- and post-test 
scores (Hotelling T (1,233) = 22.1, p = .000). Participants who received the English Class 
scenario in the pre-test performed equally well on the Roommate scenario post-test. 
Participants who received the Roommate scenario in pre-test did much worse on the 
English Class scenario post-test. This may be because the English Class scenario was 
easier. 
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Table 7. Intercorrelations of tacit knowledge measures and cognitive ability covariates. 
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Hypothesis Test 

To test the hypothesis that the experimental reflection method would improve 
performance on tacit knowledge measures more than the analytic reflection method 
(control), a GLM repeated measures ANOVA was conducted, with tacit knowledge 
vignette (CSQ) pre- and post-test as the within-subject variable and experimental 
condition, the between-subject independent variable. The results of the repeated measures 
ANOVA showed a marginally significant effect of condition on tacit knowledge post-test 
performance (Hotellings T (1, 233) = .015, p = .06). 

To examine the effect of experimental conditions on post-test scenario scores, an 
analysis of variance was conducted in which the PS2 and PS3 scores were collapsed into 
scenario pre- and post-test scores. It should be noted, however, that because the scenarios 
appear to differ in difficulty, this analysis of variance must be considered exploratory. 

The collapsed scenario post-test data violated the assumption of homogeneity of variance 
(Levene’s test F (1, 233) = 3.87, p = .05). Accordingly, a GLM univariate analysis of 
variance was conducted, with scenario post-test as the dependent variable, scenario pre¬ 
test as a covariate, and condition as the independent variable. In this test the effect of 
condition was not significant. 

The difference in post-test variance was further examined to compare the possible 
differential impact of experimental condition on variance. The experimental group had 
significantly less variance than the control group, suggesting that experimental reflection 
method in particular might account for a reduction in variance. To further explore this 
phenomenon, the relationship between the two dimensions that were scored (content and 
thoughtfulness) was compared between conditions in a series of regressions in which 
scenario content scores were regressed on scenario thoughtfulness scores in pre- and 
post-tests. In the experimental group, but not the control group, post-test thoughtfulness 
scores predicted post-test content scores (F (2, 115)= 30.84, p = .000; (3 = .282, t (1,116) 
= 5.553, p = .000), which further supports the possibility that the experimental reflection 
method may have had an impact on thought processing. 

Discussion 

This research provided a more stringent test of the experimental reflection method 
by comparing it to an analytic reflection control using practical, domain-consistent 
materials. These findings provide preliminary evidence that the combined reflection 
method improves practical problem solving, as measured by vignette performance, more 
than a closely matched analytic reflection method. 

The impact of the experimental reflection method on scenario performance 
presents a more complex picture that requires further empirical research. Consistent with 
findings in Research Effort 1, the combined method did not improve scenario post-test 
scores. However, the experimental reflection method may have had a unique effect on 
thought processing as suggested by a significant difference in post-test variance found 
between the experimental and control groups, with a reduction of variance in the 
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experimental group. Moreover, thoughtfulness scores predicted content scores in scenario 
post-test for the experimental group but not the control group. 

Scenario measures are more content-rich and, thus, require more complex 
cognition. Furthermore, responses call for the generation of solutions that involve 
articulation of tacit knowledge. Given these complexities, it is likely that more extensive 
reflection time, as well as intervening time between training and task performance for 
cognitive integration, may be needed for such methods to be beneficial. 

GENERAL DISCUSSION 

This investigation shows that the skills underlying the acquisition of tacit 
knowledge can be taught. Reflection methods derived from the theory of successful 
intelligence effectively promoted practical problem solving in research participants in 
applied settings in two distinctly different domains (i.e., military leadership and college 
life). Not only does theory-based reflection appear to be more effective than simple 
practice, as substantiated by the military leadership research (Research Effort 1), there is 
also evidence that certain types of reflection methods are more effective than others at 
promoting tacit knowledge. In the college life research (Research Effort 2), an analytic 
reflection method that prompted participants to examine and analyze domain-specific 
issues was less effective than a theory-based method that combined reflection on action 
and condition. 

In brief interventions, the relative effects of reflection methods that focus on 
condition, action, or a combination of the two aspects of experience seems to depend on 
the level of task complexity. With tacit knowledge vignettes, methods that combined 
reflection on condition and action were effective. With tacit knowledge scenarios, 
methods that focused on condition, or action aspects separately were more effective than 
a combined approach. While the evidence suggests that a combined approach has an 
effect on cognitive processing of scenario tasks, whether or not this would have resulted 
in improved practical problem solving remains an empirical question. 

Limitations 

The primary limitation of this research is that transfer of training over time was 
not assessed. Post-test assessments followed reflection interventions in a single session. 
Furthermore, the amount of time required for reflection methods to enhance practical 
problem solving over time was not evaluated. Participants were engaged in reflection 
exercises for a brief time period (30 to 40 minutes). It is possible that certain methods 
may require more time than others to be effective. Moreover, as stated previously, this 
may depend upon the nature and complexity of the task. For example, scenario measures, 
which more closely simulate the complexity of real life issues, may require more time. 

Analyses were limited in this investigation by the use of newly developed 
scenario measures, which were not equivalent. Tacit knowledge measures are domain- 
specific and consequently must be customized to the particular domain under inquiry. 
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While developing tacit knowledge measures that simulate the real world issues is labor 
intensive, this effort seems defensible when one considers that measures developed for 
assessment can be directly applied for development purposes. 

Implications 

While a variety of reflection methods have been advocated and used to enhance or 
foster the exchange of tacit knowledge in a range of workplace settings, the effectiveness 
of these methods may vary considerably depending upon how they are designed and 
implemented. This investigation is a primary step in identifying theory-based methods 
that facilitate the development and use of tacit knowledge. There is much more work to 
be done to verify the effect of reflection methods on learning and performance, identify 
that factors that moderate the relationship between reflection method and effectiveness, 
and examine the transfer of training over time. For example, in the military, differential 
effectiveness of reflection method and modes of implementation may depend on level of 
leadership. More experienced officers whose experience may no longer be appropriate in 
the current environment, may require different methods than less experienced officers 
who are just beginning to build a network of tacit knowledge in military leadership. 

CONCLUSION 

The greatest potential contribution of the theory of practical intelligence and 
previous work on tacit knowledge may be in improving our understanding of the process 
of on-the-job learning and ways to facilitate it. What we have learned from this inquiry is 
that even very brief reflection methods based on the theory of practical intelligence can 
improve practical problem solving. Further development of these and other theory based 
methods and measures promises to enhance the capacity of Military leaders to learn 
quickly and effectively from experience, a challenge that has never been more critical 
than in the current military environment. 
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Appendix Al. Military Scenario: Platoon Scenario (PS2) 

Instructions 

The following case study describes a problem that might be encountered by a platoon 
leader. You are asked to take the role of the leader described. You are also provided with 
some background information about the problem and various supplemental materials, 
such as reports or memos, that you may find useful in assessing the situation. We are 
interested in your thoughts and considerations in developing a response to this scenario. 

We understand that some individuals taking this case study will have attained a rank 
higher than PL. These individuals have been PLs at one time, and have hopefully learned 
from their experiences, no matter how long ago they occurred. These individuals should 
fill out the case study with their advice, based on their own experiences, which they 
would give a new PL for how to handle the situations presented in the case study. 

Please read through the scenario and determine the nature of the problem and what 
specific information is useful; then develop a solution. There are a series of questions at 
the end of the scenario to help you formulate a response. There are no right or wrong 
answers to these questions. We are not interested in textbook answers but rather in how 
you personally assess the situation and the response you develop to it. Please provide 
detailed and specific responses. The scenario should take 20-30 minutes to complete. 


Overview 

You are 2nd LT Pete Quandry, and have recently taken over an infantry platoon with 30 
Soldiers and 4 Bradleys. Because you have just come on board as a PL, you need to learn 
a lot about weapon systems and procedures. The former PL left nothing on paper to help 
you get oriented. 

The platoon is currently in a state of flux because PSG Joe Forte, just left. SSG Ed 
Newell, a squad leader, has been promoted from among his peers without a change in 
rank to replace him. 

Apparently, your company commander CPT Powers was very dissatisfied with the 
previous PL but had a lot of respect for the former PSG because he kept the Soldiers in 
line. He clearly has high expectations of you and the platoon and has already given you 
responsibility for a new tactical mission. You and the CPT agree that this will be a great 
opportunity for you to develop your technical skills. You also hope it will be an 
opportunity to show him that you are competent. 
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Background 

Apparently, CPT Powers found it so frustrating to work with the former PL that he often 
communicated directly with PSG Forte. PSG Forte had the reputation for being highly 
demanding and directive with the platoon. See attachment 1. 

PSG Newell knows a lot about weapon systems and procedures. You are pleased that he 
seems interested and willing to share his expertise. You will need to rely heavily on him 
to successfully accomplish the mission next week. 

The platoon has a mix of experienced and newly enlisted Soldiers. Several were in 
combat together. Attachment 2 is an early interaction with PSG Newell about the platoon. 

The training records indicate that all of the Soldiers are current on their PT and weapons 
qualifications and Newell reports that the platoon has consistently met training standards. 
However, you have some serious concern about platoon performance because during 
recent FTX, you observed that the Soldiers piled out of the vehicles and lit cigarettes 
rather than setting up a secure perimeter as their battle drill dictated. You made an on-the- 
spot correction and later counseled Newell about this. 

One week before the mission 

On Monday morning this week you discover that one of the leader books was not up to 
date in the garrison. You address this immediately with the appropriate squad leader and 
emphasize the importance of knowing the whereabouts of Soldiers at all times. Later in 
the day you discussed the issue with PSG Newell who expressed a great deal of 
frustration because he had addressed this and other issues with the squad leaders 
previously and they seem to “yes” him without following through. 

On Tuesday, there was an accident with one of the Bradleys in a training exercise. PSG 
Newell verbally reprimanded the Soldier and squad leader who were directly involved. 
See attached accident report. 
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This morning (Wednesday) you meet with PSG Newell to discuss details of the 
upcoming mission and your concerns about platoon performance. During the meeting he 
informs you that CPT Powers contacted him late in the day yesterday to inquire about 
how things were going with the mission. You were stunned to hear this because CPT 
Powers could have easily reached you yesterday at that time of day. 

You only have a few days left to motivate your troops and prepare for the mission. 

Questions : Please provide detailed and specific responses, thinking aloud about how you 
assess this situation. 

PS2-1) What problems need to be addressed in this situation? 

PS2-2) What is the single most important problem? 

PS2-3) What COA would you take to solve the problem? 

PS2-4) What specific outcome do you hope will result from the COA you have chosen? 
PS2-5) What obstacles, if any, do you anticipate to obtaining this outcome? 

PS2-6) What “lessons learned” or rules-of-thumb apply to this situation? 
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Attachment 1. An interaction with one of the NCOs 


LT Quandry: How are things going? 

SSG Pearson: Well, it’s been much better now that Forte has gone. 

LT Quandry: How so? 

SSG Pearson: Well to be honest Forte was always breathing down our necks. I mean we 
need direction but we don’t need to be told how to tie our shoelaces. Here’s a classic 
example. Once he noticed that one of my Soldiers was late one morning. This Soldier 
was usually pretty good about showing up on time and I was planning on speaking with 
him about it afterward. Before I could get to him, Forte called me aside and told me what 
to say to the Soldier, how to say it and wanted me to report back to him about how the 
Soldier responded. I rarely got a chance to handle anything by myself. It wasn’t just 
me.. .neither did anyone else. 

LT Quandry: How do you think the Soldiers responded to him? 

SSG Pearson: Well, some of them loved him, especially if he liked them. But mostly he 
intimidated them and all they really worried about was how he was going to react to 
things. It was like being a Squad Leader didn’t really matter that much to them. 
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Attachment 2 (PS2). An interaction with PSG Newell about the platoon 


LT Quandry: I would be interested in hearing any insights that you have about the 
platoon. 

PSG Newell: Well, PSG Forte was hard on us, but everyone trained to standards. 

LT Quandry: How about the personnel? 

PSG Newell: They’re good Soldiers, but two of our squad leaders, Amodio and Kane, 
don’t get along very well. It’s been going on a long time. Amodio was Forte’s favorite so 
Forte always compared the other squad leaders to him. Kane didn’t really hit it off with 
Forte, even though he had combat experience and so did Forte. 

LT Quandry: How do the other Soldiers get along? 

PSG Newell: Well to be honest with you there are a few Soldiers who have always 
seemed to be at each other’s throats. It’s been going on a long time and has never caused 
serious problems. I think because people were afraid of what Forte might do. But now 
that he’s gone.. .well it kinda worries me. 

LT Quandry: Sounds like something we need to keep an eye on. 
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U.S. ARMY ABBREVIATED GROUND ACCIDENT REPORT (AGAR) requirement control symbol 

For use of this form, sea AR 386-40 and DA Pamphlet 386-40; the proponent agency Is OCSA CS0CS-30S 



b. Coda Johnson was not wearing his helmet at the time of the accident. 








































































































































Appendix A2. Military Scenario: Platoon Scenario (PS3) 


Case Study Instructions 

The following case study describes a problem that might be encountered by a platoon 
leader. You are asked to take the role of the leader described. You are also provided with 
some background information about the problem and various supplemental materials 
such as reports or memos that you may find useful in assessing the situation. We are 
interested in your thoughts and considerations in developing a response to this scenario. 

We understand that some individuals taking this case study will have attained a rank 
higher than PL. These individuals have been PLs at one time, and have hopefully learned 
from their experiences, no matter how long ago they occurred. These individuals should 
fill out the case study with advice, based on their own experiences, they would give a 
new PL for how to handle the situations presented in the case study. 

Please read through the scenario and determine the nature of the problem, what specific 
information is useful and develop a solution. There are a series of questions at the end of 
the scenario to help you formulate a response. There are no right or wrong answers to 
these questions. We are not interested in textbook answers but rather in how you 
personally assess the situation and the response you develop to it. Please provide detailed 
and specific responses. The scenario should take 20-30 minutes to complete. 


Overview 

You are 1LT Anthony Santo, PL of a FSMC ambulance platoon with 5 wheeled- 
ambulance squads. Your platoon has been deployed overseas for three months. There has 
been no activity, although conflict in the region is brewing and the need for emergency 
medical services could occur at any time. 


It has been difficult to keep the troops motivated this period of deployment. The 
environment is extremely hot and buggy. You sense a gradual decline in morale in many 
of the units, not just your own. 


Your PSG, Tim Loyola, has been having a lot of difficulty with SSG Lewis, an 
ambulance squad leader in your platoon. You have a close working relationship with PSG 
Loyola, whose judgment in general you respect and trust. However, while he usually 
exhibits a high level of restraint and composure when dealing with personnel discipline 
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issues, increasingly you have noticed that he has been losing his temper, particularly in 
dealing with SSG Lewis. 

Background 

Loyola complains that Lewis isn’t getting his job done properly and drinks too much. 

He finds it totally unacceptable that Lewis is not more conscientious about following 
SOPs. Loyola believes firmly in adherence to procedures and regulations. He thinks that 
something should be done about it. Although you agree that maintaining standards in the 
field is important, you wonder if he is sometimes overreacting to what seem to be minor 
infractions. 

On more than one occasion Loyola observed Soldiers performing tasks that really should 
be handled only by Lewis. When asked about this, they say, “our SSG trusts us” or “he’s 
asked us to help him out.” The last time it happened Loyola insisted on knowing where 
Lewis was. The Soldiers said that they didn’t know. Later Lewis told Loyola that he 
wasn’t feeling well that morning. 

SSG Lewis’ record of past work performance is mostly good. His rescue efforts in 
combat are practically legendary. When he was a CPL, he extracted a Soldier who was 
impaled on a steel pole and administered emergency care, which kept him alive until he 
reached the treatment facility at BSA. The PL that he served under during that time is 
now CPT of your medical company. CPT Mahan still remembers Lewis and even asks 
about him from time to time. However, Lewis has had more than his share of “counseling 
sessions” for minor infractions mostly associated with preventative maintenance checks 
and services over the past few years. And you do recall seeing a letter of reprimand 
(attachment 1) in his file for disorderly conduct some time ago. 

You are aware of “personal history” between PSG Loyola and SSG Lewis. Apparently 
they were good friends but had a falling our several years ago prior to Loyola’s 
promotion to PSG. 
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SSG Lewis is very well liked in the platoon; he is quick witted and entertaining. It seems 
that the Soldiers in his squad trust him and will do almost anything for him. 

Your CO CDR, CPT Mahan, is primarily concerned with ends rather than means. As long 
as the job gets done, he is not too concerned with official policy. He takes a hands-off 
approach and feels that any good PL should be able to handle personnel problems within 
his platoon. He also has an extremely liberal attitude concerning alcohol and has been 
very lenient about enforcing alcohol policies in the past. He believes that “Soldiers need 
to unwind” and he seems to recall fondly his own youthful bar room adventures that were 
a source of good fun and camaraderie. 

Shortly before your unit was deployed, there was a very serious alcohol related accident 
that resulted in significant damage to emergency service equipment in another medical 
company. You remember receiving the battalion commander’s memo on the Army’s 
substance abuse prevention policy (attachment 2) at that time. 


Recent events 

Late in the day yesterday SSG Lewis approaches you and asks to speak with you about 
problems that he is having with PSG Loyola. You told him that you could not meet with 
him at that time and would get back to him about it. 


It is morning, and you are walking through camp when you come across SSG Lewis and 

PSG Loyola in the middle of an argument: 

PSG Loyola (forcefully): And what’s up with these sloppy supply records? Can’t 
you keep anything straight? 

SSG Lewis (calmly): I am sorry they’re out of order, here I’ll double check them 
and... 

PSG Loyola (shouting): Is that alcohol I smell on your breath? Huh? Drunk on 
duty! I am going to nail you for this! 

SSG Lewis (defensively): I am stone sober! I drank until late last night, off duty. I 
am sober this morning... 

PSG Loyola (shouting): Bull! If you didn’t drink so much maybe you could keep 
things straight around here! 

SSG Lewis (shouting): What the hell is your problem? You don’t treat other squad 
leaders like this. 
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PSG Loyola (shouting): Like what? Huh? How do I treat you? You must think 
I’m stupid... 

You can see that SSG Lewis’ eyes are blood shot, but he does not appear to be drunk 
otherwise. The argument is escalating. 
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Attachment 1: PS 3 


DEPARTMENT OF THE ARMY 
XXX MEDICAL BATTALION 
UNIT #XXX 
APO AP XXXXX-XXXX 

XXXX-XX 14 April, 2005 

MEMO FOR: CPL John Lewis 
From: XXX MED BN 
Subject: Letter of Reprimand 

1. Investigation has shown that you, CPL John Lewis, did, on or about December 18, 
1999, at 129 Sentinel St. Norfolk, VA disturb the peace and engage in drunk and 
disorderly conduct, in violation of Article 134 of the Uniform Code of Military Justice 
(UCMJ). This offense is so serious that, had I elected to punish you under the provisions 
of Article 15 of the UCMJ, you could have been subject to incarceration or involuntary 
separation from the United States Army. 

2. You are hereby reprimanded. Your actions have brought discredit upon yourself and 
require me to seriously question your judgment and sense of responsibility. Drunk and 
disorderly conduct is a serious offense which reflects badly not only on you as a person 
but as a member and representative of the Unites States Army. 

3.1 will not tolerate this type of behavior from a member of my unit. Be advised, should 
I be made aware of any future information concerning this type of behavior, I will take 
swift action, possibly resulting in your punishment under the provisions of the UCMJ and 
possibly your involuntary separation from the United States Army. 

4. Examine your career objectives and determine which course you will follow. It’s up to 
you. 

5. You will acknowledge receipt of this reprimand below. If you wish, you may attach a 
written statement or additional documents to this reprimand which will be attached to this 
document in file. If you choose to do so, such attachments must reach my office by 15 
January, 2000. 

LTC Paul Jarmin, 

Commander, XXX MED BN 
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Attachment 2. PS 3 


DEPARTMENT OF THE ARMY 
XXX MEDICAL BATTALION 
UNIT #XXX 
APO AP XXXXX-XXXX 

XXXX-XX 25 April, 2005 

MEMORANDUM FOR BATTALION LEADERSHIP 

SUBJECT: Alcohol Awareness 

1. References: AR 600-85 Army Substance Abuse Program (ASAP) 

2. Most of you are aware of the recent alcohol-related accident that resulted in serious 
injury to Army personnel and irreparable damage to medical equipment in our BN. In the 
aftermath of this unfortunate accident, I direct your attention to Army policy and 
procedures aimed at preventing alcohol abuse in the workplace. Outlined below are key 
aspects of the policy. I urge you to review AR 600-85 Army Substance Abuse Program 
(ASAP) in full and be prepared to fully implement this program as necessary. 

3. Alcohol Policies and Controls 

a. We will keep the workplace alcohol free. Also, alcohol will not be the center of 
attention at Army functions. 

b. Misconduct resulting from drinking alcohol, or impairment while on duty will not be 
tolerated. 

c. Leaders will ensure that subordinates are held responsible for their actions and are 
aware of alcohol abuse and its consequences. 

d. Leaders will refer Soldiers for screening, treatment, or prevention training if they know 
that Soldiers are abusing alcohol. 

e. Leaders are encouraged to do surprise inspections in the unit, not only to ensure 
alcohol is not present on duty, but also to promote safety and good order and discipline. 

f. If leaders suspect alcohol abuse, they must confront the suspected Soldier regardless of 
rank or performance or conduct. 

g. Even if a Soldier refers him/herself for treatment, he or she is still responsible for 
his/her actions. Furthermore, if treatment fails, he or she must be removed from the 
Army. 

h. If a Soldier is identified as having a problem with alcohol, he or she must successfully 
complete ASAP education or a rehabilitation program in order to remain in the Army. 

4. Alcohol Sanctions 

a. Soldiers may be punished under UCMJ (Articles 111 and/or 112) or separated from the 
Army if they are involved in serious alcohol related offenses (more than two DWI 
convictions in a 1-year period). 

b. Any Soldier who performs duties with a blood alcohol level of .05 percent or above 
will be subject to UCMJ and administrative disciplinary action. The only exception to 
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this is if the Soldier was unaware of the duties that needed to be performed at the time the 
Soldier became impaired. 

c. Detoxification and appropriate treatment will be provided to any Soldier diagnosed as 
alcohol dependent. 

Jim T. Swagart 
LTC, XX 
Commanding 
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The scenarios and questions you have read are designed to present our participants with a complex mix of problems that a 
military leader may face. The problems can be solved with multiple possible solutions and outcomes, some of which would be more 
effective than others. The purpose of our scoring procedure is to determine and rate the extent to which leaders can identify the 
problems in a given situation, create an appropriate solution for addressing those problems, and anticipate the effects of his or her 
decisions and actions. 



PSG - Platoon Sergeant _ 

PVT - Private _ 

SOP - Standard Operating Procedure 
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Long-Term Problems with Cause and Effects 

• A long-term problem that demonstrates an understanding of factors that may result from it 







Place a check mark in the appropriate box and use this scoring guide below to 
assign an overall score for this question 

Is the problem Is it an Immediate Is it just an Is it a Long-Term + Is it just a Long- 

Inappropriate! +Causality problem! Immediate problem? Causality problem? Term problem? 

(If checked move down 
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7 points - Respondent lists more than two problems but less than half of them are immediate or immediate with cause and effects 

8 points - Respondent lists two problems; at least one is immediate and none are inappropriate 

*Add 1 pt. for each additional immediate problem provided 

9 points - Respondent lists two problems; at least one is immediate with cause and effects and none are inappropriate 
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Place a check mark in the appropriate box and use this scoring guide below to 
assign an overall score for this question 
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3. What COA would you take to solve this problem? 
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Please rate each outcome by circling the appropriate numbers the table below, 
and following instructions to compute a total sum score 
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The scenarios and questions you have read are designed to present our participants with a complex mix of problems that a 
military leader may face. The problems can be solved with multiple possible solutions and outcomes, some of which would be more 
effective than others. The purpose of our scoring procedure is to determine and rate the extent to which leaders can identify the 
problems in a given situation, create an appropriate solution for addressing those problems, and anticipate the effects of his or her 
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Long-Term Problems with Cause and Effects 

A long-term problem that demonstrates an understanding of factors that may result from it. 






Place a check mark in the appropriate box and use this scoring guide below to 
assign an overall score for this question 

Is the problem Is it an Immediate Is it just an Is it a Long-Term + Is it just a Long- 

Inappropriate! +Causality problem? Immediate problem? Causality problem? Term problem? 

(If checked move down 
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7 points - Respondent lists more than two problems but less than half of them are immediate or immediate with cause and effects 

8 points - Respondent lists two problems; at least one is immediate and none are inappropriate 

*Add 1 pt. for each additional immediate problem provided 

9 points - Respondent lists two problems; at least one is immediate with cause and effects and none are inappropriate 

*Add 1 pt. for each additional immediate problem provided SCORE:_ 
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2. What is the single most important problem? 
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2 points - Respondent provides a combination of long-term and immediate problems but none are single most important problems 

3 points - Respondent provides one or more immediate problem(s) but aren’t single most important problem(s) 

4 points - Respondent lists multiple problems including one single most important problem 

5 points - Respondent provides one or more single most important problems SCORE:_ 










The response to Q3 may specify one or more courses of action (COAs). Please rate each CO A using the table below. You will need to 
refer to Q2 for these ratings. 
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4. What specific outcome do you hope will result from the COA you have chosen? 
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Please rate each outcome by circling the appropriate numbers the table below, 
and following instructions to compute a total sum score 


d 



-H (N 
4) 4) 


C/3 C/3 

X X 


C/3 

X 


O O O 




* 

* 



co 


SCORE: 


















Appendix C. Procedures for Developing College Life Scenarios 
Generating the contextual framework 

Nine resident advisors from a liberal arts college in the MidAtlantic were interviewed for one 
hour via the telephone. They were asked to describe experiences that taught them important lessons 
about college life. From these interviews, researchers culled common themes and developed four 
sample scenario drafts that featured situations that may be encountered in college life 

Developing the final two scenarios 

Two final scenarios were created that integrated some aspects of the four original drafts. They 
were designed to provide enough information so that there might be multiple ways to interpret the 
nature of the problem and construct a potential solution. Open-ended questions were added to measure 
the cognitive components of tacit knowledge. 

Both scenarios were piloted with a group of eight upperclassmen that provided written 
responses to scenarios questions and, then, discussed their reactions to scenario description, in terms 
of realism, and questions, in terms of understandability. Minor revisions were made to improve upon 
realism and expand the range of potential responses. 
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Appendix Dl. College Life Scenario: Roommate 

Instructions 

The following case study describes problems that may be encountered by a college student. You are 
asked to take the role of the student described. You are also provided with some background 
information about the problems presented that you may find useful in assessing the situation. Please 
read through the scenario and determine the nature of the problems presented and what specific 
information is important to consider, and then develop a response to questions presented at the end of 
the scenario. 

Please explain your thoughts and considerations used in constructing a response. There are no right or 
wrong answers to these questions. We are interested in your personal assessment of and response to 
the situation rather than a textbook type answer. Please provide responses that are thoughtful, detailed 
and specific. This scenario should take 20-30 minutes to complete. 

Overview 

You are a college freshman living in the residence halls. Your roommate, Jamie, was randomly 
assigned to live with you at the beginning of the academic year, and the two of you have gotten along 
quite well up until this point. But, tensions are rising, not only because of midterm exams next week, 
but also because lately Jamie hasn’t been her normal self. This semester, more often she seems down. 


Lately Jamie has been neglecting her studies and spends more time partying on the weeknights. You 
are aware of all of this, want to be supportive as a friend, but you are not sure what to do. 


Background 

Jamie seemed like the kind of person who could be your best friend from the moment that you met. 
The move to college had been particularly difficult for you because you had never been away from 
your family for an extended period of time. The friendship that you formed with Jamie was a big help. 
She introduced you to new people, made you feel comfortable in new social situations, and always 
helped you adapt to your new surroundings. Now that you’ve entered the second semester, Jamie 
seems less and less willing to study or hang out like she used to. Lately, she prefers drinking and going 
to parties. Jamie also seems to be sleeping very little, and often you find it difficult to sleep because 
she is so loud. 


In your many attempts to understand Jamie’s change of character, she has never identified a single 
problem, but occasionally mentions how much happier she was in high school, and, now that she’s in 
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college, how confused she feels about what to do with her life. You have wondered what’s going on 
with her family because she rarely seems to speak with them. When you ask about her family, she says 
everything is fine and doesn’t elaborate. 

Recent Events 

You have begun to worry more and more about Jamie. You notice that she missed class twice this 
week. You also notice that she has skipped out on her intramural soccer practices. You tried to 
encourage her to join you in doing things that you typically enjoy together, like going to the movies, 
but Jamie is completely uninterested. 


Academics have been more challenging for both you and Jamie this semester. The Professors seemed 
to have taken it easy on the freshmen during the first semester so that they could get used to college 
life. However, course work this semester seems much more demanding and both you AND Jamie are 
feeling more stressed by it. You are taking a rigorous class together and, because Jamie keeps missing 
class, she often asks you to help her with her homework. 


The situation with Jamie is really taking its toll on you. You often find it difficult to concentrate in 
class. Rather than thinking about the lesson, you find yourself trying to figure out what to do about 
Jamie. And, not only is your lack of concentration affecting the time you spend in the classroom, it is 
also affecting your homework. You also find yourself tired during the day because Jamie has kept you 
up late the night before. 


Because of your concern, you sent an e-mail about it to one of your high school friends, Sue, who 
attends a different college than you. Her response was as follows: 

Hey, 

Sorry to hear about your roommate. I don’t think you should get too worried though. Lots of 
my friends are freaking out right now. School is just getting tougher. The teachers think we can 
handle it all now and are laying it on us. I mean, 1 didn ’t see one of my friends for a week 
because she had to finish a class project. Your roommate’s behavior change doesn ’t mean she 
has any serious problems. It’s just natural that your mood goes up or down based on the 
amount of work you have to do. So maybe your roommate might be down right now but it’s 
probably only temporary. 

Don’t worry too much about it and have fun this year! 

Sue 
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Next week is midterms. Jamie doesn’t seem to be studying at all. Last night she got pretty drunk and 
you ended up staying up to take care of her again. You are very worried about doing well next week 
especially because of this situation with Jamie. 

Questions : Please provide detailed and specific responses indicating how you assess and would 
respond to this situation. 

1) What problems need to be addressed in this situation? 

2) What is the single most important problem? 

3) To resolve the problems in this situation effectively, please describe in detail what actions you 
would take. 

4) To resolve this situation effectively, what would you be sure NOT to do? 

5) What specific outcomes do you hope will result from the course of action you have chosen? 

6) What obstacles, if any, do you anticipate when obtaining these outcomes? 
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Appendix D2. College Life Scenario: English Class 

Instructions 

The following case study describes problems that may be encountered by a college student. You are 
asked to take the role of the student described. You are also provided with some background 
information about the problems presented that you may find useful in assessing the situation. Please 
read through the scenario and determine the nature of the problems presented, what specific 
information is important to consider, and develop a response to questions at the end of the scenario. 

Please explain the thoughts and considerations you used in constructing a response. There are no right 
or wrong answers to these questions. We are interested in your personal assessment of and response to 
the situation rather than a textbook-type answer. Please provide responses that are thoughtful, detailed, 
and specific. This scenario should take 20-30 minutes to complete. 


Overview 

You are a college sophomore who is enrolled in a rather difficult upper-level English class. You are 
used to getting rather high marks on your papers, and you have already put a lot of time and effort into 
your English class. But, the last two papers that have been graded and returned to you have had lower 
grades than you anticipated, leading you to question your English professor’s grading strategy. There 
is an upcoming group project, (for which every group member will receive the same grade), worth 
15% of your entire semester grade, on which you hope to do well to make up for the poor grades on 
your papers. 


Background Information 

Dr. Asher, your English professor, has been with the university for close to 30 years and is highly 
respected on campus for his literary achievements as well as for his excellent and very demanding 
teaching. Yet, it is known among the students that Dr. Asher “plays his favorites” in the class. You 
have heard that Dr. Asher tends to favor students who have taken another of his classes prior to this 
one. This, however, is your first time in one of his classes. 


You got a “C+“ on your first paper. You were disappointed by the grade, not only because of all the 
time you had put into the paper, but also because you had never received lower than a B+ on a paper in 
your previous two college English classes. Furthermore, the comments on the paper were rather 
general and the reasons why you received such a poor grade were not apparent. 
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You decided to approach Dr. Asher to ask for help with the next paper, which is due the next week. 
His only advice was to start the paper early and seek help at the Writing Help Center. 

TIMELINE 

Monday, October 9 th : You took the advice Dr. Asher offered, began the paper early, and reviewed it 
with an assistant in the “Writing Help Center” on campus. You turned the paper in today, confident 
that this paper was better than the last. 

Today in class, you also received assignments for the group project, due on October 25th—two and a 
half weeks from now. Unfortunately, students didn’t get to pick fellow group members. Students 
assigned to your group agreed to meet on Friday. 

Friday, October 13 th : During your first real group meeting, you all picked the parts of the project for 
which you would be responsible. One of your group mates, Catherine, who seems to work as hard as 
you do, also wants to get a good grade in the class. She has not had any problems with her previous 
papers. The other person, Ron, seems to care, but his ideas are outlandish and at times very 
impractical. You also don’t like his egotistical “know it all” attitude. He seems to think he is a more 
advanced student because he has taken an earlier class with Dr. Asher. You have also observed that he 
gets along very well with Dr Asher. You all agreed to meet on Wednesday to begin to put your 
individual pieces together. 

Wednesday, October 18 th : At this meeting, you and Catherine discovered that you don’t like Ron’s 
work. It looks like it he didn’t put in the effort you felt was necessary. Since his ideas for the project 
were not very practical, and his written work seemed disorganized and confusing, it felt like you were 
going around in circles trying to decide what to do with his section. You began to feel that he was 
almost not worth having on the team. You and Catherine dismissed most of his ideas and decided to 
go with your own. Ron did not say much to either of you after the meeting. The following night you 
and Catherine spent hours working on Ron’s section without including Ron. 

Today, Friday, October 20 th : At the beginning of class. Dr. Asher takes a moment to discuss the group 
project. He indicates that he is “seriously concerned” because he has heard some people complain 
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about being left out of their groups. “This project,” he states, “is not just about getting a good grade. 
It’s about learning how to work as a team.” He ends with a chilling remark about how one’s actions in 
this project could, in fact, affect their overall class grade. 

At the end of class he hands back the second paper. You are stunned to see another “C+” written 
across the top. There are as few comments as the 1st paper. When Dr. Asher hands back Ron’s paper, 
you are surprised to see a “B+” written at the top. 

You have one more week before this project is due (October 27th), and you are supposed to meet with 
Ron and Catherine later on tonight to work on the project. You are not sure what to do. 

Questions : Please provide detailed and specific responses, indicating how you assess and would 
respond to this situation. 

1) What problems need to be addressed in this situation? 

2) What is the single most important problem? 

3) To resolve the problems in this situation effectively, please describe in detail what actions you 
would take. 

4) To resolve this situation effectively, what would you be sure NOT to do? 

5) What specific outcomes do you hope will result from the course of action you have chosen? 

6) What obstacles, if any, do you anticipate when obtaining these outcomes? 
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Appendix El. Scoring Rubric: Roommate Scenario 


College Life Scenarios: Guidelines for Scoring 
Introduction 

The College Roommate Scenario is made up of six open-ended questions designed to assess 
experience-based (tacit) knowledge in college life. In a nutshell, we need to assess the extent to which 
responses reflect knowledge that experienced college students may have acquired about college life. 
We have collected data from college life “experts,” i.e. college seniors/graduates, for comparison 
purposes. 

Your job is to rate responses to these open ended questions on a five-point scale. Below you will find 
detailed guidelines for scoring each question. The first three questions involve rating on two broad 
dimensions, content and thoughtfulness. The final three questions are rated based on the content 
dimension only. 

Please review the attached Roommate Scenario followed by open-ended questions that respondents 
answered. Then read the scoring guideline for each question fully before you begin to rate responses. 

Feel free to contact us should you have any difficulties or questions. Cynthia Matthew can be reached 
at Cynthia.matthew@vale.edu or (436-1544), and Cassandra Nichols at Cassandra.Nichols@yale.edu 
or (432-3858). 

Thank you and have fun! 
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Scoring Guide: Roommate Scenario 


Question 1; Content “What problems need to be addressed in this situation?” 

This question is rated on a five-point scale that considers the extent to which the responses capture the 
knowledge conveyed in expert responses, which is summarized below. Basically, the more problems 
on the expert list that are included in the response, the higher the score. Below is a summary of expert 
responses and rating scale guidelines including examples. 


Expert Response Summarv 

Five Point Rating Scale Guidelines 

1. Jamie ’ s problems: 

1. Response indicates a problem or comment 

a) May be symptomatic of deeper 

that is not included in the expert summarv 

mental/emotional issues 

or does not answer the question, (e.g.. 

i. Depression 

“Missing classes”). 

ii. Drinking 


iii. Family problems 

2. Response covers one of the problem areas 

b) Are compromising her health, grades 

listed by experts, (e.g., “ Jamie’s partying, 

and academic career 

not sleeping, and skipping classes, and 

c) May lead to future safety, health, and 

not doing any work”). 

career problems. 


2. The effect on me 

3. Response covers two of the problem areas 

a) Are compromising your health, grades 

listed by experts, (e.g., “The fact that 

and academic career 

Jamie is taking away my study time too. 

i. Loss of sleep 

Why Jamie has changed. What is 

ii. Preoccupation with Jamie’s 

wrong?”). 

problem 


iii. Loss of concentration on my 

4. Response covers three of the problem 

studies. 

areas listed by experts, (e.g., “What is 

b) May lead to future health and academic 

wrong with Jamie? Jamie is losing her 

problems. 

focus in school and needs help getting 

c) Loss of /miss her friendship 

back on track. My lack of sleep and 

3. The effect on friendship 

midterms are coming up ”). 

a) Straining it 


b) Poor communication 

5. Response covers four or more of the 

c) Behavior is not respectful 

problems areas listed by experts, (e.g., 

d) Jamie is taking advantage of me by 

“Why Jamie has suddenly changed her 

using my school work 

behavior. Whether family problems, 

e) Behavior could lead to resentment and 

stress, or workload relate to her change in 

eventual breakdown in friendship 

behavior. What should be done to help 

4. How to help Jamie 

Jamie (professionally) so you both can 

5. How to take care of myself 

live with each other? Rules for the room 

a) Study in the library 

so Jamie doesn ’t keep you up late ”). 
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Question 1: Thoughtfulness “What problems need to be addressed in this situation?” 


This five-point rating scale assesses the complexity of thought in the response. Ratings are based on 
the extent to which the response reflects an understanding of potential “causes and effects” associated 
with the problem situation. In other words, knowledge of possible underlying factors that may have 
given rise to the problem, and/or the future consequences if the problem is not addressed. Below are 
rating scale guidelines including examples. 

5-point rating scale guidelines: 

1. Response does not suggest a cause and effect association. 

a. It does not ask why the problem may exist or consider possible consequences that may 
result. 

b. It is completely off topic 

c. It merely provides a recitation of the problems listed in the scenario. 

(e.g., “Attending and studying for class. I don’t think partying and drinking is a problem.''’) 

2. Response suggests a cause and effect linkage that focuses on a superficial or tangential aspect 
of the problem. 

(e.g., “Jamie could get hurt one night when she is drunk...The problem could get worse and ruin 
our friendship.”) 

3. Response suggests a cause and effect that is central to the problem situation and stated rather 
generally. It does not suggest specific reasons or consequences. 

(e.g., “Why is she not feeling like herself? Why is she not going to class? Why is she drinking and 
partying so much?") 

4. Response suggests a central cause and effect linkage and suggests relevant underlying 
problems or potential consequences. 

(e.g., “What is making Jamie feel so down? Is there something going wrong in her family? Is she 
depressed/having depression? Why does she not want to go to class? Does she feel all alone and 
feel like no one cares about her?") 

5. Response suggests a central cause and effect that includes statements about relevant underlying 
problems, possible consequences of the problem if they are not addressed, and how a solution 
might be found. 

(e.g., “The fact of her staying up all night because it is keeping you up too. Temporary mood 
swings or not, she’s being inconsiderate. Also, that you have to take care of her. She gets drunk, 
she should learn from it. Babying someone won’t teach them anything. What’s going on in her 
head. Although she has no obligation to tell you.") 
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Question 2: Content “What is the single most important problem?” 

This rating scale is based on the extent to which the response captures expert responses in terms of 
content and specificity. Expert responses are broken down into: major problem areas and related 
subcategories, which are specific examples of the broader category (in bold type); and less significant 
problem areas, and related subcategories (not in bold type). Below is a summary of expert responses 
and rating scale guidelines including examples. 


Expert Response Summary 

1. Jamie’s problems: 

a. Possible psychological problem: 
depression, drinking and/or family 
problems 

b. Possible emotional instability 

i. Compromising her health, 
grades and academic career. 

2. The effect on me: 

a. My personal well-being 

i. Loss of sleep 

b. My overall academic performance 

i. My midterm performance 

3. The effect on our friendship 

a. Poor communication 

b. Behavior is not respectful 

c. Jamie is taking advantage of me 

4. How to address both problems 1 & 2 

a. Trying to find a balance between 
saving our friendship and protecting 
myself. 

b. Reacting appropriately to the situation 


Five-point rating scale guidelines 

1. Response focuses on a problem that does 
not appear in the expert summary or does 
not answer the question, (e.g., “There is 
not a single most important problem. All 
have affected her in different ways and all 
need to be dealt with”). 

2. Response selects a sub-category which is 
included in the expert summary but is a 
not a major subcategory, and, therefore, 
not in bold type , (e.g., “Studying for 
midterms"). 

3. Response indicates one of the major 
categories listed by the experts in bold 
type but very generally stated, i.e. without 
subcategory information, (e.g., “Her 
affecting my study habits”). 

4. The response indicates one of the major 
categories and any additional subcategory 
information that suggests a more 
developed understanding of the major 
problem area, (e.g., “Jamie is not being 
respectful to my needs, especially around 
the crucial weeks of midterms. She should 
take into consideration how her actions 
affect me ”). 

5. The response integrates more than one 

major category and suggests a developed 
understanding of how problem areas are 
linked, (e.g., “Probably the drinking 
because it affects Jamie’s schoolwork, 
sleep patterns, and probably moods, and 
it also affects my studying and sleeping 
time as well”). _ 









Question 2: Thoughtfulness “What is the single most important problem?” 


This five-point rating scale assesses the complexity of thought in the response. Ratings are based on 
the extent to which the response reflects an understanding of potential “causes and effects” associated 
with the problem situation. In other words, knowledge of possible underlying factors that may have 
given rise to the problem, and/or the future consequences, if the problem is not addressed. Below are 
rating scale guidelines including examples. 

5-point rating scale guidelines: 

1. Response does not suggest a cause and effect association. 

a. It does ask why the problem may exist or consider possible consequences that may 
result. 

b. It is completely off topic 

c. It merely provides a recitation of the problems listed in the scenario. 

(e.g., “ Jamie’s behavior”) 

2. Response suggests a cause and effect linkage that focuses on a superficial or tangential aspect 
of the problem, (e.g., “That she doesn’t study and midterms are in a week”). 

3. Response suggests a cause and effect that is central to the problem situation and stated rather 
generally. It does not suggest specific reasons or consequences, (e.g., “Can she handle being in 
college?”). 

4. Response suggests a central cause and effect linkage and suggests relevant underlying 
problems or potential consequences, (e.g., “Jamie cutting into my time and my success (college 
is not cheap)”). 

5. Response suggests a central cause and effect that includes statements about relevant underlying 
problems, possible consequences if the problem if they are not addressed, and how a solution 
might be found, (e.g., “The most important thing is getting Jamie back on track. Not only will 
that help her, but it will make life a little easier for me”). - 


Question 3: Content “To resolve the problems in this situation effectively, please describe in detail 
what actions you would take?” 

The five-point rating scale assesses the extent to which the participant’s course of action (COA) 
overlaps with the expert responses in terms of both content and sequence . The summary of expert 
responses has been organized into 4 general “types” of COA’s mentioned by the experts that have 
been placed in the order that they should appear from first to last. Below is a summary of expert 
responses and rating scale guidelines including examples. 
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Expert Response Summary 

1) Speak with Jamie about the problem 

a) Content 

i) Reflect your honest 
observations and concerns 

ii) Address how her behavior is 
affecting you (e.g.,“You need 
to express your need for more 
sleep and for her not to depend 
on you for school work, etc.”) 

b) Approach 

i) Be supportive—let her 
know you are there for her and 
ask if you can help. 

ii) Confront her; be stem, but 
make sure she understands your 
doing this because you care. 

2) Change your own “enabling” behavior 

a) Suggest some sort of acceptable 
compromise for both Jamie and myself. 

i) Ask her to be quiet when 
she comes in late at night 

ii) Work out a fairer 
arrangement regarding the 
difficult class you both take so 
that you aren’t doing all the 
work (e.g.“Set up an 
arrangement such that she and I 
each attend alternate classes and 
fill each other in later on.”) 

b) Attempt to draw Jamie’s attention 
to the problem indirectly by acting 
independently of her 

i) Go to the library to study 

ii) Go to meals without her 

3) Seek and/or suggest higher authority 
assistance for Jamie 

a) Residence Advisor 

b) College counselor 

c) Psychologist 

d) Dean. 

e) Jamie’s parents 

4) If the situation doesn’t change or gets 
worse remove yourself by moving out or 
requesting another roommate. 


5-point rating scale guidelines 

1. The response does not contain a COA in 
the expert summary or answer the question 
(e.g., “7 would not single out the problem, 1 
would just wait and see what happens. 
Chances are she is just trying to enjoy 
college as much as possible.”) 

2. The response contains only one COA 
mentioned by the experts OR several COA’s 
that do not correspond to the experts’ 
sequence (e.g., “Possibly report her to 
security so they would issue an alcohol 
citation. Possibly talk to her 
parents/family/friends about her. Show an 
active interest in her well-being.”) 

3. The response contains one type of COA 
mentioned by the experts and covers several 
different sub-categories under that COA, 
reflecting more detailed understanding of the 
steps to take. 

4. The response provides 2 types of COA’s 
mentioned by the experts in the correct 
sequence (e.g., “7 would talk to her and 
explain that I am really worried about her. If 
this didn’t work, I’d speak to her parents 
about the matter. ”) 

5. The response provides 3-4 types of 
COA’s mentioned by the experts that are 
generally in the correct sequence, 

e.g., “7 would sit her down and explain to 
her that first 1 am her friend and 1 love 
her to death. That I am here for her when 
she needs me. I will be on her side not 
matter what. But 1 feel she needs help 
and I am here to help her study more and 
party less and not to flunk out. Or I 
would send her to a psychologist.” 
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Question 3: Thoughtfulness “To resolve the problems in this situation effectively, please describe in 
detail what actions you would take?” 

Ratings for this section assess the extent to which the responses reflect an understanding of the 
relationship between one’s actions and the potential outcome(s) of these actions. A high score should 
be assigned to responses that address one aspect of the scenario in terms of how particular actions may 
result in particular outcomes. Responses should address a relevant sequence of action and action 
outcomes or reactions. Below is a summary of expert responses and rating scale guidelines including 
examples. 

Note: It may be tempting to assign a high rating to a response that addresses multiple aspects of the scenario in a general 
way. Please avoid doing this! 

Five-Point Rating Scale Guidelines: 

1. The response is simple and does not link action to outcome, (e.g., “7 would say I can’t look 
after you all the time. You need to get your act together. But I wouldn ’t be that worried in the first 
place ”). 

2. There is suggestion of an action-outcome linkage, but it is superficial, (e.g., “I would not take 
it upon myself to help her. I won’t let her take me down too. 1 will do well in school even if it 
means that I don’t spend time to help her’’). 

3. The response discusses an important but general action-outcome linkage, (e.g., “7 would sit her 
down, schedule out an hour or two, and just ask her what the hell is going on. Talk things 
through. Establish the problem ’’). 

4. The response discusses more than one important and specific action-outcome linkage 
associated with a specific problem. It begins a sequence and is specific, 

e.g., “7 would first try to talk to my roommate nicely, but if nothing changes, 1 would firmly 
tell her that her behavior is extremely distracting and that something must be done because 
I cannot afford to lose sleep and baby-sit her. ” 

5. The response produces a complex and coherent sequence of actions and outcomes to a specific 
problem. It is as if the respondent is actually putting himself/herself in the situation and can 
picture how the situation might unfold, 

e.g., “7b resolve the situation, I would start by telling Jamie that I care for her but that her 
behavior is having a negative impact on me and is unacceptable. 1 would tell Jamie that 
she can talk to me about her problems, but if the problems are so big that she cannot solve 
them on her own, she should see a counselor. ” 
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Question 4 “To resolve this situation effectively, what would you be sure NOT to do?” 

This question is rated on a five-point scale that considers the extent to which the responses capture the 
knowledge conveyed in expert responses, which is summarized below. Basically, the more concepts 
on the expert list that are captured in the response, the higher the score. 


Expert Response Summary 

1. When speaking to her 

a. Timing is important 

i. Don’t start out by 

suggesting counseling or 
threatening to move out. 

b. Approach is important. 

i. Don’t get angry or 
accusatory. 

ii. Avoid fighting. 

2. Contacting others about the problem 

a. Don’t gossip about her problems. 

b. Don’t go to an authority figure 
before letting Jamie know. 

3. Managing your involvement 

a. Don’t enable Jamie further by 
joining her when she parties and 
drinks. 

b. Don’t become too emotional or 
overly involved with Jamie’s 
situation. 

4. Do not ignore Jamie or the situation 


Five-Point Rating Scale Guidelines 

1. Response indicates a problem or comment 
that is not included in the expert summary or 
does not answer the question, (e.g., “Get her 
to hate me for stupid reasons. But she can 
hate more all she wants if it’s because 1 made 
her and 1 study and do good on exams, it’s 
ok”). 

2. Response covers one of the concepts listed by 
the experts, (e.g., “Ignore it, it would just get 
worse"). 

3. Response covers two of the concepts listed by 
the experts, 

(e.g., “What you shouldn’t do is let it 
go. You should at least try and help 
your roommate out. Another thing 
would be not to get angry and yell at 
your roommate—you are just making 
the problems worse”). 

4. Response covers three of the concepts listed 
above, 

e.g., “I definitely would not be 
confrontational with Jamie, that 
sounds like the last thing she needs. 1 
would try not to avoid her though 
either. I would not nag at her about 
work, since that isn ’t going to better 
anything. ” 

5. Response covers four or more of the concepts 
listed by the experts, 

e.g., “I wouldn’t ignore Jamie or 
make the problem worse by being mad 
at her because it would add another 
stress to worry about. 1 wouldn’t 
encourage her change in attitude from 
her previous self because it is messing 
up her academics. I wouldn’t stop 
being there for her when she is drunk, 
but would try to explain I can’t be 
doing that all the time.” 
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Question 5 “What specific outcomes do you hope will result from the course of action you have 
chosen?” 

This rating scale measures the extent to which the participant’s expected outcomes overlap with the 
expert responses. The summary of responses is organized into 4 general “categories” of expected 
outcomes mentioned by the experts. The general expected outcomes highlighted in bold are the most 
relevant. The non-bold expected outcomes (the “subcategories”) are important, but only partially 
correct. 


Ex pert Response Summary 

1. Jamie will open up to you 

a) Recognize the problem 

b) Recognize its impact on you 

2. Jamie will change her behavior 

a) She will drink less 

b) She will come to class more 

c) She will be a better roommate 

d) Get good grades 

3. I will be less distracted 

a) Get more sleep 

b) Study more 

c) Get good grades 

4. The friendship will be restored. 

a) Study together 

b) Party together 


Five-Point Rating Scale Guidelines 

1. Response does not include an outcome in the 
expert summary, but may contain a 
subcategory (e.g., “She will drink less”). 

2. Response contains one major outcome in the 
expert summary but does not provide any 
additional subcategory information (e.g., 
“Jamie will open up to you”). 

3. Response contains one major outcome 
category and subcategories that more fully 
capture the concept. 

e.g., “Jamie will calm down and take her 
actions elsewhere. She will realize how 
she is harming her good friend/roomie. 

She might also realize that she needs to 
shape up for herself and not just her 
roomie. ” 

Response provides more than one major 
category indicated in the expert summary, 
provides little or no subcategory information. 
e.g. “I hope that if we can open up and talk to 
each other that both our concerns can be 
addressed and we can work together to find a 
solution that works for both of us. ” 

5. Response provides more than one major 
category of outcomes and includes 
subcategory information that more fully 
captures the concept. 

e.g., “7 hope that Jamie will resolve 
whatever problems she is having and pay 
more attention to her schoolwork and not 
party so much. 1 would want to see her 
happier and more considerate o me-like 
when try to do homework or go to sleep. I 
would hope that we would become closer 
as friends-the way we were when we first 


4. 
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Question 6 “What obstacles, if any,.do you anticipate when obtaining these outcomes?” 

This rating scale measures the extent to which the participant’s expected obstacles overlap with the 
expert responses. The summary of responses is organized into 5 general “categories” of expected 
obstacles mentioned by the experts. The major expected obstacles highlighted in bold are the most 
relevant. The non-bold expected obstacles (the “subcategories”) are important, but only partially 
correct. 


Expert Summary 

1. Talking about it won’t work 

a. Jamie won’t be receptive/willing to 
talk 

b. She may not follow through on 
agreed upon arrangements. 

2. Talking about it could make it worse 

a. Jamie will become angry with me. 

b. She may guilt trip me. 

c. Talking with precipitate an 
outpouring of emotion that 
overwhelms you both. 

d. If you talk to her and she doesn’t 
change you may become more 
frustrated, less understanding and 
more stressed. 

e. The friendship will fall apart. 

3. Jamie will be resistant and unwilling to 
change 

4. Jamie may already be out of control and 
unable to change 

5. Poor overall academic performances. 

a. Poor midterm exam performance. 


Five-Point Rating Scale Guidelines 

1. Response lists no obstacles, obstacles not 
indicated by the experts, or is not 
answering the question (e.g., “Trying to 
get your friend back to normal again. 
Have everything settled between the 
two”). 

2. Response vaguely addresses one of the 
five major obstacles in the expert 
summary, or lists a subcategory (e.g., 
“Her self-esteem problems”). 

3. Response addresses one of the major 
obstacles more fully, including its 
subcategories (e.g., “Lots of resistance, 
maybe even some hostility directed at 
you. She may also want to try to avoid 
you as much as possible”). 

4. The response addresses more than one 
major obstacle (e.g, “Her resistance to 
change. A more solid block between 
us”). 

5. Response addresses more than one major 
obstacle, and fully captures them by 
including subcategory information, 

e.g., “If she doesn’t realize that she’s doing 
anything wrong, she may have trouble 
stopping her actions. Her mood may be 
depression, getting her to open up could be 
hard. Therapy may be good. Listen to her 
talk without being judgmental. Getting her to 
admit that she’s acting different. 
Hostility/Resentment towards you. ” 
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Appendix E2. Scoring Rubric: English Class Scenario 

College Life Scenarios: Guidelines for Scoring 

Introduction 

The English Class Scenario is made up of six open-ended questions designed to assess experience- 
based (tacit) knowledge in college life. In a nutshell, we need to assess the extent to which responses 
reflect knowledge that experienced college students may have acquired in college life. We have 
collected data from college life “experts,” i.e. college seniors/graduates, for comparison purposes. 

Your job is to rate responses to these open-ended questions on a five-point scale. Below you will find 
detailed guidelines for scoring each question. Questions one and three involve rating on two broad 
dimensions, content and thoughtfulness. Questions two, four, five and six are rated based on the 
content dimension only. 

Please review the attached English Class Scenario followed by the open-ended questions that the 
respondents answered. Then read the scoring guideline for each question fully before you begin to rate 
the responses. 

Feel free to contact us should you have any difficulties or questions. Cynthia Matthew can be reached 
at Cvnthia.matthew@ vale.edu or (436-1544), and Cassandra Nichols at Cassandra.Nichols @yale.edu 
or (432-3858). 

Thank you and have fun! 


Important Tips for Rating this Scenario: 

1) It is essential that you review the expert responses carefully, including their major categories 
and subcategories. 

2) Use your judgment when a response contains examples that are not found in the expert 
responses. If you still feel that it deserves a high score, please feel free to rate it as such. Just 
remember to always be consistent with your ratings. 

3) There will be responses that you might not feel are deserving of content-related high score 
(particularly in questions 4, 5 and 6), but the rating scate states that they must be rated highly. 
We believe that the thoughtfulness scales for questions 1 and 3 will compensate for this, so 
please be consistent with the five-point scale guidelines. 

4) The writing can be poor and confusing at times. Please try to infer as best as you can. 
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Scoring Guide: English Class Scenario 
Question 1: Content “What problems need to be addressed in this situation?” 

This question is rated on a five-point scale that considers the extent to which the responses capture the 
knowledge conveyed in expert responses, which is summarized below. Basically, the more the 
response suggests problems associated with the major problems areas on the expert list, the higher the 
score. Below is a summary of expert responses and rating scale guidelines including examples. _ 


1 . 


Expert Response Summary 

My individual performance in the class: 

a) My work may not be good enough. I 
may need to make improvements. 

b) I need more information about Dr. 
Asher’s expectations and his 
assessment of my work to understand 
why putting in more effort did not 
improve my grade. 


2 . 


How to do well in the group project 
a) How to include Ron in the group 
project. 

How to deal with the quality of 
Ron’s work. 

How to incorporate Ron’s work in 
the group project without 
compromising quality. 

Improving your working relationship 
with Ron. He may have complained 
to Dr. Asher. 

Dr. Asher’s grading policies are not clear, 
a) Dr. Asher is not responsive or 
helpful. 

4. Dr. Asher’s grading policies may not be fair, 
a) Dr. Asher may be playing favorites. 


b) 

c) 


d) 


3. 


Five Point Ratine Scale Guidelines 

1. Response indicates a problem or comment that 
is not included in the expert summary, or is not 
relevant in your judgment (e.g., “When are we 
going to find time to do this?”). 

2. Response vaguely covers one of the major 
problem areas, or vaguely covers a problem 
relevant in your judgment, (e.g., “Address Ron 
and tell him to put more effort into the 
assignment”). 

3. Response suggests clear and specific problems 
associated with one of the major problem areas 
listed, or a major problem you feel is relevant, 
(e.g., “The problems not being addressed are: 
Ron working with his group more frequently 
and also communication in the group”). 

4. Response suggests problems associated with 
two of the major problem areas listed, or two 
problems relevant in your judgment, (e.g., 

“Why Dr. Asher seems to like Ron more than 
the other students. Catherine and I should work 
with Ron instead of dismissing his ideas”). 

5. Response suggests problems associated with 

three or more of the major problems areas listed, 
or three or more problems areas relevant in your 
judgment (e.g., “Why am I doing poorly on my 
part? Is Dr. Asher really favoring some 
students? How can I tell Ron his work needs a 
little improvement?”). _ 
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Question 1: Thoughtfulness “What problems need to be addressed in this situation?” 


This five-point rating scale assesses the extent to which the response reflects insight into the problems. 
In other words, ratings should be based on the extent to which the response reflects knowledge of 
possible underlying factors that may have given rise to the problem, and/or the future consequences if 
the problem is not addressed. Below are rating scale guidelines including examples. 

5-point rating scale guidelines: 

1. Response does not suggest any insight. 

a. It does not ask why the problem may exist or consider possible consequences that may 
result. 

b. It is completely off topic. 

c. It merely provides a recitation of the problems listed in the scenario. 

(e.g., “The teacher’s grading. Ron’s attitude and connection with Dr. Asher.”) 

2. Response suggests insight but focuses on a superficial or tangential aspect of the problem. 

(e.g., “Ron doesn’t do any work and I keep getting C+’s on my papers. C’s don’t bother me 
but to get them all the time can be kind of annoying.”) 

3. Response demonstrates insight into problems that are central to the situation but stated rather 
generally. It does not suggest specific reasons or consequences. 

(e.g., “Why am I not receiving higher grades on my papers? How can we include Ron in the project? 
How can I try to become more friendly with Mr. Asher?”) 

4. Response demonstrates insight into problems that are central to the situation and suggests 
relevant underlying problems or potential consequences. 

. (e.g., “It needs to be addressed that we are not working as a group and we need to be and that we need 
to come up with some plan on how we’re going to make everyone’s pieces fit together.”) 

5. Response demonstrates insight into the problems that are central to the situation suggests and 
includes statements about relevant underlying problems, possible consequences of the problem 
if they are not addressed, and how a solution might be found. 

(e.g., “Ron needs to understand that while his ideas make sense to him, they are confusing to 
the other group members; and they don’t go along with the work of the other 2 people. He 
needs to be convinced that the 3 people need to compromise. Catherine and I should apologize 
for working without him and say that we really need to figure out a way that everyone can 
work on the project and come out OK; maybe incorporating more of Ron’s ideas if he can 
explain them better.”) 
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Question 2: Content “What is the single most important problem?” 

This rating scale is based on the extent to which the response captures expert responses in terms of 
content and specificity. Expert responses are broken down into: major problem areas and related 
subcategories, which are specific examples of the broader category (in bold type); and less significant 
problem areas, and related subcategories (not in bold type). 


Expert Response Summary 

1. How to get a better grade on individual 
work 

a. How to obtain more information 
about Dr. Asher’s expectations and 
his assessment of your work. 

i. How to elicit constructive 
input on your work from Dr. 
Asher. 

ii. Understand the discrepancy 
between the amount of work 
you are putting in and your 
grades. 

2. How to get a good grade on the group 
project. 

a. How to include Ron’s work in a 
productive way. 

b. Ron is not contributing enough. 

c. How to repair the group dynamic. 

3. The possibility that Dr. Asher may grade 
unfairly. 

a. He may favor Ron. 

b. He may favor previous versus new 
students. 

c. He may play favorites. 

4. Communication gap 

a. How to more effectively 
communicate with Prof. Asher. 

b. How to effectively communicate 
with group members. 


Five-point rating scale guidelines 

1. Response focuses on a problem that 
does not appear in the expert summary, is 
not relevant in your judgment, or does 
not answer the question, (e.g., “Finish 
project”). 

2. Response selects a sub-category 
which is included in the expert summary 
but is not a major subcategory, and, 
therefore, not in bold type, (e.g., ‘That 
you are not getting along as a team”). 

3. Response indicates one of the major 
categories listed above in bold type but 
very generally stated, i.e. without 
subcategory information, (e.g., “How can 
I do better in the class?”). 

4. The response indicates one of the 
major categories and any additional 
subcategory information that suggests a 
more developed understanding of the 
major problem area. 

e.g., “My grade is the most important 
problem. If I was doing everything I 
could and still receiving aC+I would 
want to know why.” 

5. The response integrates more than 
one major category and suggests a 
developed understanding of how problem 
areas are linked. 

e.g., “The teacher is reinforcing Ron’s 
lackadaisical work effort, and Ron is 
responding to hurt the group project. 
Because it is not only affecting the grade 
but it is making my effort and work seem 
less important than just being a favorite.” 
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)uestion 3: Content 


“To resolve the problems in this situation effectively, please describe in detail 
what actions you would take.” 


The five-point rating scale assesses the extent to which the participant’s course of action (COA) 
contains the major areas suggested by the experts in terms of content o nly. The summary of expert 
responses has been organized into 4 general “types” of COA’s. Ratings are simply based on the 
number of types of COA’s suggested by the response. Please use your judgment; if a response 
contains an example that is not mentioned in the expert summary, but in your view is 
representative of a major category, rate it as such._ 


Expert Response Summary I 5-point rating scale guidelines 


1. Discuss your concerns about your paper grades 
with Dr. Asher. 

a) Schedule a meeting with Dr. Asher and 
explain your concerns in detail. 

b) Ask him to provide more specific 
feedback to help you understand how to 
improve your papers. 

2. Talk to Catherine about your concerns about your 
paper grades and group project. 

a) Discuss how you can include Ron’s work 
in the project. 

b) Ask her if you can read her paper. 

3. Take steps to improve working together as a 
group. 

a) Talk to Catherine. 

b) Talk to Ron. 

c) Take it up jointly as a group. 

d) Ask for Prof. Asher’s help. 

e) Make an effort to engage Ron 

1. Apologize for dismissing his 
ideas 

2. Make him feel welcome. 

3. Appease him 

f) Ask for Ron’s help 

1. Ask for his input on paper 
grades. 

2. Let him know how important it 
is for you to get a good grade on 
this project. 

g) Work on producing a quality group 
project. 

1. Explain your concerns about 
Ron’s ideas. 

2. Explain your ideas to Ron and 
convince him of their value. 

3. Edit Ron’s work 

4. Seek the help of an outside official. 

a) Seek the advice of a former English 
rofessor. 


Response does not contain a COA in the expert 
summary, contains a COA that is not relevant in 
your judgment, or does not answer the question 
(e.g., “Knowing that Ron always gets good grades 
and is well-liked by the teachers, I would do the 
project however he said it should be done”). 

Response vaguely addresses one type of COA 
listed on the expert summary, or vaguely 
addresses one COA relevant in your judgment, 
(e.g., “I would sit down and discuss the problems 
with the group then act accordingly”). 

Response clearly and in greater detail addresses 
one type of COA listed by the experts, or one 
COA that is relevant in your judgment, 
e.g., “Discuss different sections of the project. 

Try to help Ron with his section so it works 
together with the rest of the project, and try to get 
Ron to take his work seriously.” 

Response suggests 2 types of COA’s listed by the 
experts, or 2 COA’s relevant in your judgment, 
e.g., “I would talk to Ron and tell him that he 
needs to put more of an effort in. If I continued to 
get bad grades I would have to go to someone 
higher up in the department and ask for their 
advice.” 

Response provides 3 or more types of COA’s 
listed by the experts, or 3 or more COA’s relevant 
in your judgment. 

e.g., “I would make an appointment with Dr. 
Asher before I met with the group. I would find 
out exactly what was wrong with my paper. Then 
I would meet with my group because that is a 
separate situation and handle things responsibly 
with them. Then I would make another 
appointment with the writing center for more help 
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on how to fix my paper.” 


b) Ask the head of the writing department to 
grade the paper. 

c) Complain to the Head of the Department 
or Dean of Students. 

d) Visit the Writing Center for more 
assistance. 







Question 3: Thoughtfulness “To resolve the problems in this situation effectively, 

please describe in detail what actions you would take?” 

Ratings for this section assess the extent to which the responses reflect insight into the relationship 
between one’s actions and the potential impact of them on the problems in the scenario (i.e., action 
and action-outcome). A high score should be assigned to responses that suggest an action sequence 
that addresses important aspects of the problem situation. Bear in mind the number of aspects 
addressed is not important. What is important is the thoughtfulness that can be inferred from the 
response. 

Note: It might be tempting to assign a high rating to a response that addresses multiple aspects of the 
scenario in a general way. Please avoid doing this! 

Five-Point Rating Scale Guidelines: 

1. Response is simple, does not show any insight into the relationship between actions and their impact or 
outcomes on the problem, or just doesn’t answer the question, (e.g., “I would tell Catherine and Ron that 
the project is all of ours and not everyone is doing their work”). 

2. Response reflects insight into the relationship between action and potential action-outcomes, but applied to 
a superficial aspect of the problem situation, 

e.g., “I would talk to my professor and tell him that Ron is not putting in the effort like everyone 
else. I would also say I wrote his paper. And how I deserve a better grade like Ron’s because I 
write the same way. I deserve a B+.” 

3. Response reflects insight into the relationship between action and action-outcomes of an important aspect of 
the problem situation stated generally , (e.g., “I would talk to Ron and Catherine and hope that we could 
reach some conclusion on how to make this project work. Also how we’re going to work as a group”). 

4. Response reflects insight into the relationship between action and action-outcomes of important aspects of 
the problem in greater detail . It begins a sequence and is specific, 

e.g., “I would first attempt to discuss my feelings regarding his grading policy with the professor, 

highlighting the main reasons I feel it is unfair. If nothing changed, I would try to find other 
students who agreed with me and take it to the administration.” 

5. Response produces a complex and coherent sequence of actions and outcomes to important aspects of the 
problem situation. It is as if the respondent is actually putting himself/herself in the situation and can 
picture how it might unfold, 

e.g., “Instead of dividing up the work on the project, do it as a group, with all three members working 
together. That way you can weed out a lot of Ron’s bad ideas while still including him on the 
project. With the papers, go to the professor with your last paper and go over with him how you 
could have done better.” 
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Question 4 “To resolve this situation effectively, what would you be sure NOT to do?” 

This question is rated on a five-point scale that considers the extent to which the responses capture the 
knowledge conveyed in expert responses, which is summarized below. Basically, the more concepts 
on the expert list that are suggested in the response, the higher the score. *Bear in mind that the 
response may not use the term (s) “Don’t” or “Do Not” in responses about “what not to do”.* 


Expert Response Summar 


Five-Point Ratine Scale Guidelines 


1. Do not anger Professor Asher. 

a. Approach him in a non-confrontational 
manner. 

b. Do not accuse him of favoritism (unless 
you are absolutely sure and as a last 
resort.) 

c. Do not go to the Dean of Students or 
another official right away. Do this only 
when all other avenues for resolving the 
problem are exhausted. 

2. Do not anger or upset Ron. 

a. Do not say he is one of Dr. Asher’s 
favorites. 

b. Do not say that his work is sloppy and 
yours is better. 

c. Do not explicitly ask him if he spoke to 
Dr. Asher. 

3. Do not let yourself get too angry with anyone 
involved including yourself. 

4. Do not mismanage the project 

a. Do not continue to exclude Ron on the 
group project. 

b. Do not take complete control of group 
project. 

c. Do not give Ron complete freedom on 
the project. 

5. Do not ignore the problems 

a. Problems associated grades on your 
written work. 

b. Problems associated with group project. 


1. Response indicates a problem or comment that 
is not included in the expert summary, is not 
relevant in your judgment, or does not answer 
the question, (e.g., “Not to put it off until the 
day before”). 

2. Response vaguely covers one of the concepts 
listed, or one concept relevant in your judgment, 
(e.g., “I would not make Ron feel out-casted”). 

3. Response covers in greater detail one of the 
concepts listed, or one concept relevant in your 
judgment. 

e.g., “I wouldn’t come out and say I know why 
Ron received this grade. I also wouldn’t say it 
was because he was one of his favorites. I 
wouldn’t accuse him of giving a better grade 
based on favorites.” 

4. Response covers two of the concepts listed, or 
two concepts relevant in your judgment, (e.g., 
“Make the teacher mad by accusing him of 
being biased. And also no accusing Ron, but be 
sincere about it”). 

5. Response covers three or more of the concepts 
listed, or three or more concepts relevant in your 
judgment, 

e.g., “Accuse Ron of being lazy or trying to 
slide through class because he already knows 
Dr. Asher. Get angry and Dr. Asher and 
directly accuse him of playing favorites. 
Continue to work as a group, the same as 
before. Neglect to look into the English Dept, 
olicies surrounding grading.” 
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Question 5: “What specific outcomes do you hope will result from the course of action you have 
chosen?” 


The five-point rating scale assesses the extent to which the response includes major areas on the expert 
list in terms of content and specificity . The summary of expert responses has been organized into 4 
general “types” of outcomes with multiple subcategories. Ratings are simply based on the number of 
types suggested in the response and the degree to which they more fully capture the concept by 
including subcategory information. 


Expert Response Summary 

1. Talking with Dr. Asher will be effective. 

a) You will gain greater understanding of the 
expectations. 

b) Dr. Asher will provide constructive 
criticism of your work 

c) Dr. Asher will recognize your efforts and 
commitment to excel. 

d) Dr. Asher will accept that you don’t think 
the grade is fair, and will negotiate a fairer 
grade with you. 

2. Taking steps to include Ron in the group project 
will yield positive results. 

a) Ron will be cooperative. 

b) You will achieve a compromise that 
involves everyone’s involvement. 

c) A group project in which all members feel 
satisfied. 

d) Your group will obtain a good grade on 
the project. 

e) Ron will recognize the value of your & 
Catherine’s work on the project. 

3. Seeking outside help will be beneficial. 

a) You will gain an understanding of Dr. 
Asher’s behavior and get advice. 

b) If you need to go to a higher authority, she 
or he will resolve the problem. 

4. You will gain personally from the experience 

a) Obtain a good individual grade. 

b) Learn from Catherine & Ron. 

c) Become a better writer. 


Five-Point Rating Scale Guidelines 

1. Response does not include an outcome in the 
expert summary, it is not relevant, or it does not 
answer the question, (e.g., “Switching classes”). 

2. Response contains one outcome in the expert 
summary, or one outcome relevant in your 
judgment, stated generally , (e.g., “I will learn how 
to work better with people”). 

3. Response provides at least one outcome in the 
expert summary, or one relevant in your 
judgment, that includes subcategories and more 
fully captures the concept, (e.g., “Hopefully, he 
will reevaluate my writing and I will receive a 
better grade, or he will give me guidance with my 
future papers”). 

4. Response contains two outcomes in the expert 
summary, or two outcomes relevant in your 
judgment, stated generally , (e.g., “Hopefully the 
professor will ease my essay scores, and we’ll get 
a good grade on the project”). 

5. Response provides two outcomes in the expert 
summary, or two outcomes relevant in your 
judgment, that include subcategorv information. 
more fully capturing the concept. 

e.g., “I hope the professor would realize if he’s 
being unfair and I would hope he would realize 
that Ron is not putting much effort into the 
project. I would hope I would gain a better 
understanding of what the professor expects from 
my work in order to ameliorate my future 
assignments. I would hope Ron would realize that 
he needs to make as much of an effort as I am and 
that we would get a better grade.” 
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Question 6 “What obstacles, if any, do you anticipate when obtaining these outcomes?” 

The five-point rating scale assesses the extent to which the response includes major areas on the expert 
list in terms of content and specificity . Expert responses are grouped into 4 general “types” of 
outcomes with more specific subcategories. Ratings are simply based on the number of types 
suggested in the response and the degree to which the response is clear, specific and reflects a more 
developed understanding of the nature of the obstacle. 


Expert Summary 

1. Speaking to Dr. Asher won’t help. 

a. He will not provide you with 
information to help you understand 
his requirements and how to improve 
your writing 

b. He may not be willing to negotiate 
your grade. 

c. He may be unfair in his grading 
practices. 

2. Ron will not be responsive to your efforts to 
include him. 

a. He may be angry and resistant to 
working together. 

b. He may not accept your ideas for the 
project. 

c. It will be difficult to find a 
compromise that doesn’t anger Ron. 

d. He may continue to feel excluded and 
possibly talk to Dr. Asher 

e. The burden of the work for the group 
project will continue to fall on you 
and Catherine. 

f. You may become angry with Ron and 
do or say the wrong thing. 

3. You and Catherine may not agree on how to 
approach the situation, which could result in 
additional strained relations. 

4. Including Ron’s work in the group project 
may make comprise the quality of work. 

5. Seeking help from outside or a higher 
authority will not be beneficial. 


Five-Point Rating Scale Guidelines 

1. Response lists no obstacles, irrelevant obstacles, 
obstacles not indicated by the experts, or is not 
answering the question (e.g., “I wouldn’t 
anticipate any outcomes. It is what it is”). 

Response vaguely addresses one of the five 
obstacles listed, or vaguely addresses one obstacle 
relevant in your judgment (e.g., “Dr. Asher may 
not provide the information I am looking for”). 

3. Response addresses one of the obstacles listed in 
the expert summary (or one relevant in your 
judgment) clearly and more specifically reflecting 
a more developed response. 

e.g., “Dealing with Ron. I would have to deal with 
him rejecting our ideas and only liking his own. I 
would also have to be especially nice to him so he 
doesn’t tattle to the professor again.” 

4. The response vaguely addresses two obstacles 
listed, or two obstacles you find relevant. 

e.g, “If Ron spoke to Dr. Asher, then Dr. Asher might 
be predisposed to not give us a good grade on the 
project. That Ron will not do his part with respect to 
completing the project”). 

5. Response addresses two or more obstacles listed 
more clearly and specifically reflecting a more 
developed response . 

e.g., “It might be hard to get Ron to work 
cooperatively with us again, considering he obviously 
told Dr. Asher he was excluded (unless the same thing 
is happening in another group too). Dr. Asher might 
not be willing to tell me what I need to get a better 
grade on my papers. And he might be defensive if I 
say I feel I deserved a higher grade.” 


2 . 


62 










Appendix FI. Sample reflection interventions: Reflection on Condition 

Improving Problem Solving Skills (Battalion Commander) 

In this brief exercise, we would like to turn your attention to how you come to understand the 
leadership problems you face on the job. You will read questions about the leadership problems you 
encountered in the vignettes and case study, and also about problems you have encountered in your 
work. These questions may seem abstract, so they are explained below. The first two questions are: 

“What is the problem that must be solved? What goal do you intend to reach by solving it?” 

In these questions we are asking you to differentiate between a problem and a goal . A problem is a 
conflict that must be resolved, a challenge that must be overcome, or a balance that must be achieved 
in order to accomplish a particular goal . Using terms from the FM 101-5, a goal is analogous to a 
“mission” in that stating a goal involves stating the purpose for action. A problem is analogous to a 
“key task” in that problems represent the conditions that must be met for mission success or goal 
accomplishment. So, when we ask you to identify the problem in a vignette or case study, we are not 
asking you to provide us with a goal. We are asking you to state your goal and to identify what 
difficult thing must be accomplished in order to meet that goal. 

Consider the example below. What is the problem that must be solved in this situation? 

You are a battalion commander and it is the end of your first battle at a major externally-evaluated 
training exercise, during which your unit revealed some major shortcomings. During the AAR, the 
Chief Evaluator is highly critical of the battalion and dwells on all the negative things your unit did 
that day. You carefully record all of the negative observations, but you know full well that the 
battalion also did some very positive things that day. What should you do? 

Your goal in this situation might be to make sure that the Chief Evaluator recognizes the positive 
things your battalion did in the training exercise. In this case, the problem that must be solved is to 
find a way to assert yourself with the Chief Evaluator without undermining your credibility. 

An alternative goal might be to improve your battalion’s performance in future training exercises. In 
this case, the problem that must be solved is to establish and implement a set of methods that will lead 
to increased unit effectiveness and readiness. 

Another question you will see often is the following: 

“Describe the factors you considered when choosing your goal and determining the problem that must 
be solved (e.g., doctrine, personal values, assumptions about army culture, procedures, and personnel, 
knowledge based on previous experiences)” 

By asking you to describe the factors listed above, we are trying to get you thinking about how you 
determine your goals and identify which problems to solve on the job. There are multiple ways to 
decide on goals and identify the critical problems to solve. These ways differ in how effectively they 
lead to a problem solution. By understanding how you choose a particular goal and how you come to 


63 



identify the problems you face, you can better understand why you take the actions you do, and 
perhaps improve the effectiveness of your problem solving. 

Using the example above, if you defined the problem as one of asserting yourself with the Chief 
Evaluator, you might describe the factors as follows: 

“I assumed that the Chief Evaluator’s on-sided evaluation would have a negative effect on unit morale. 
I know from past experience that bad morale undermines unit cohesion and leadership capability. 

Also, I assumed that by asserting myself I would show my unit that I recognize their strengths. This 
would also improve morale.” 

If you defined the problem as one of establishing and implementing methods for increasing unit 
effectiveness and readiness, you might describe the factors as follows: 

“I assumed that the Chief Evaluator just had a negative style and that my unit would not take the on- 
sidedness of the evaluation too seriously. I have learned from past experience that you learn a lot more 
from hearing about your mistakes than from hearing praise. Evaluations are not about building self¬ 
esteem.” 

We understand that you did not necessarily think this analytically when you examined the vignettes 
and case study earlier in today’s session. Much of this kind of thinking is automatic and outside of our 
awareness. However, just as a golfer or baseball player must examine his swing in detail in order to 
improve it, we suspect that a leader must examine his reasons and thinking behind problem solving in 
order to improve his decision-making capability. 

On the following page, you will encounter the questions described above and you will be asked to 
think about, well, how you think. Please answer these questions thoughtfully and thoroughly, using the 
digital recorders. You will have approximately 40 minutes to complete the exercise. If at any point 
during this exercise you have questions about what we are asking you to do, please raise your hand 
and someone will assist you. 

Think about the first vignette you read (reprinted below). 

B3. You are a new battalion commander and one of your most important and challenging tasks is to 
establish the training priorities for your unit. While everything looks important and you would like to 
meet every possible contingency, you also realize that you do not have the time or resources to “do it 
all.” 

COG-la) What is the problem that must be solved in this vignette? 

COG-lb) What goal do you intend to reach by solving it? 

COG-2) Describe the factors you considered when choosing your goal and determining the problem 
that must be solved, (e.g., doctrine, personal values, assumptions about army culture, procedures, and 
personnel, knowledge based on previous experiences) 

COG-3a) What is an alternative goal you could have? 
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C0G-3b) What problem must be solved if you wish to reach this goal? 

COG-4) Describe the factors that played a role in choosing this goal and determining the problem you 
* just described. 

COG-5) Imagine that your goal is to increase the amount of training resources available to you. One 
problem that must be solved is determining a strategy for re-allocating existing resources to meet the 
current training needs. Describe the factors that would play a role in choosing this goal and 
determining problem identification. 

Think about the leadership problems in the case study you read. 

(You may flip back to it to review your answers, if necessary.) 

COG-6) Describe the factors you considered when you chose your goal and identified the most 
important problem in the case study, (e.g., doctrine, personal values, assumptions about army culture, 
procedures, and personnel, knowledge based on previous experiences) 

COG-7) Imagine you had identified one of the secondary problems as the main problem. What goal 
would you accomplish by solving this problem as if it was the most critical? 

COG-8) Describe the factors that would play a role in choosing this goal and determining this problem 
identification. 

Think about a leadership problem that challenged you to re-examine your goals and assumptions. 

COG-9) Briefly describe the problem situation. What was your initial goal, and the problem you had 
to solve to achieve it? 

COG-10) Describe the factors that played a role in choosing your goal and determining your problem 
identification. 

COG-11) Why didn’t your initial understanding of the problem result in effective action? 

COG-12) What factors did you have to consider to change your understanding of the problem? 
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Appendix F2. Sample reflection interventions: Reflection on Action 


Improving Problem Solving Skills (Battalion Commander) 

In this brief exercise, we would like to demonstrate how reflecting on the unexpected outcomes of our 
actions can provide information that improves our problem-solving capability. On the next few pages, 
you will be presented with the answers that other military personnel gave to one of the vignettes and 
the case study you just completed. You will be asked to reflect on the differences between your 
answers and the answers provided by others and to reflect on how these differences might have come 
about. 

One way to reflect on how differences in actions come about is to ask questions about where a 
person’s decisions to act come from. As you well know, different people can encounter the same 
situation but interpret it very differently. These different people therefore act very differently. With the 
questions we ask in this exercise, we are trying to turn your attention to how you come to understand 
the leadership problems you face on the job and how your understanding affects your decision to act. 
These questions may seem abstract, so they are explained below. The first two questions are: 

“What is the problem that must be solved? What goal do you intend to reach by solving it?” 

In these questions we are asking you to differentiate between a problem and a goal . A problem is a 
conflict that must be resolved, a challenge that must be overcome, or a balance that must be achieved 
in order to accomplish a particular goal . Using terms from the FM 101-5, a goal is analogous to a 
“mission” in that stating a goal involves stating the purpose for action. A problem is analogous to a 
“key task” in that problems represent the conditions that must be met for mission success or goal 
accomplishment. So, when we ask you to identify the problem in a vignette or case study, we are not 
asking you to provide us with a goal. We are asking you to state your goal and to identify what 
difficult thing must be accomplished in order to meet that goal. 

Consider the example below. What is the problem that must be solved in this situation? 

You are a battalion commander and it is the end of your first battle at a major externally-evaluated 
training exercise, during which your unit revealed some major shortcomings. During the AAR, the 
Chief Evaluator is highly critical of the battalion and dwells on all the negative things your unit did 
that day. You carefully record all of the negative observations, but you know full well that the 
battalion also did some very positive things that day. What should you do? 

Your goal in this situation might be to make sure that the Chief Evaluator recognizes the positive 
things your battalion did in the training exercise. In this case, the problem that must be solved is to 
find a way to assert yourself with the Chief Evaluator without undermining your credibility. 

An alternative goal might be to improve your battalion’s performance in future training exercises. In 
this case, the problem that must be solved is to establish and implement a set of methods that will lead 
to increased unit effectiveness and readiness. 
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Another way to reflect on how differences in actions come about is to ask questions about how 
different actions result in different outcomes. As you well know, different people can have similar 
interpretations of a situation and yet take very different actions to handle it. You will also be asked 
questions about the actions you chose versus those of the other military personnel, and what outcomes 
you would expect to result from these actions. We are interested in specific answers to these questions, 
as opposed to something like “I would reach my goal.” 

We understand that you did not necessarily think this analytically when you examined the vignettes 
and case study earlier in today’s session. Much of this kind of thinking is automatic and outside of our 
awareness. However, just as a golfer or baseball player must examine his swing in detail in order to 
improve it, we suspect that a leader must examine his reasons and thinking behind problem solving in 
order to improve his decision-making capability. 

On the following three pages, you will encounter the questions described above and you will be asked 
to reflect on how having different goals and identifying different problems lead to different problem¬ 
solving strategies. Please answer these questions thoughtfully and thoroughly, using the digital 
recorders. You will have approximately 40 minutes to complete the exercise. If at any point during this 
exercise you have questions about what we are asking you to do, please raise your hand and someone 
will assist you. 


Think about the first vignette you read. 

It is reprinted below with the average ratings given by 59 AWC students designated as expert battalion 
commanders. 

B3. You are a new battalion commander and one of your most important and challenging tasks is to 
establish the training priorities for your unit. While everything looks important and you would like to 
meet every possible contingency, you also realize that you do not have the time or resources to “do it 
all.” Rate the following strategies for how effective they would be in helping you establish your 
priorities. 

_8_Study the brigade's training schedule. 

_8_Talk to the brigade S-2, S-3, and CSM to verify your understanding of the brigade commander's 

training focus. 

_7_Schedule meetings to discuss training with each of your staff members during your first week 

of command. 

_8_Explain your goals and your plans for the battalion very clearly to your officers and staff. 

_4_Assess the tactical and technical competence of your Soldiers individually by giving them 

formal and informal tests. 

_6_Rely on the assessments made by the previous battalion commander. 


67 


7 _Select three to five upcoming missions (based on the brigade training plan) to focus your 

Soldiers' energy on. 

8 _Before doing anything, make sure you understand the commander’s intent two levels up. 

8_Soon after taking command, visit each staff section's shop and get a full briefing on their 

operations. 

.8_Talk to the brigade commander to determine his training priorities. 

1234 5678 9 


Extremely Somewhat Neither Bad Somewhat Extremely 

Bad Bad Nor Good Good Good 


Note: See Part II for Overview and Instructions on Vignettes 

In this vignette, my ratings differed from the AWC students’ on the following options (please indicate 
with a check in the space provided on the left of the option): 

_Study the brigade's training schedule. __ 

_Talk to the brigade S-2, S-3, and CSM ... _ 

_Schedule meetings to discuss training ... _ 

_Explain your goals and your plans ... _ 

_Assess the tactical and technical competence ... _ 

_Rely on the assessments made ... _ 

_Select three to five upcoming missions ... _ 

_Before doing anything, make sure you understand_ 

_Soon after taking command, visit... _ 

_Talk to the brigade commander ... _ 

My ratings differed from the AWC students’ ratings in the following ways (Please indicate the degree 
of difference by writing the appropriate abbreviation to the right of the relevant response option.): 
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(MH) Much higher (5 or more points difference) 

(H) Higher (3-4 points difference) 

(SH) Slightly higher (1-2 points difference) 

(SL) Slightly lower (1-2 points difference) 

(L) Lower (3-4 points difference) 

(ML) Much lower (5 or more points difference) 

My ratings differed from the students’ ratings perhaps because we had different goals in mind, and 
identified a different problem to be solved... 

REF-la) What do you think is the problem that must be solved in this vignette? 

REF-lb) What goal do you intend to reach by solving it? 

REF-2a) What do you think the AWC students’ goal might have been? 

REF-2b) What problem do you think they intended to solve? 


Perhaps the students and I had similar goals and identified a similar problem, but we preferred 
different actions... 

REF-4) What outcome do you feel your actions would achieve? 

REF-5) What outcome do you feel the students’ actions would achieve? 

Think about the leadership problems in the case study you read. 

(You may flip back to it to review your answers, if necessary.) 

One of the questions from the case study is reprinted below with the average answer from 23 NCOs. 


What CO A would you take to solve the problem? 

Approach CPT Powers about coming directly to me with his concerns. Call a meeting with PSG 
Newell and counsel him to “crack the whip” with the squad leaders. Take a direct role in supervising 
training exercises. 

REF-6) How does your COA differ from the one preferred by the NCOs? 

REF-7) What problem do you suppose the NCOs identified as most important in order to prefer the 
COA they did? 

REF-8) What outcome do you feel the NCOs COA would achieve? 

REF-9) How does the expected outcome of the NCO COA differ from that of your own COA? 
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Think about a leadership problem that challenged you to re-examine your goals and assumptions, 
where your actions didn ’t result in the outcome you expected. 

REF-10) Briefly describe the problem situation. What was your initial goal, and the problem you had 
to solve to achieve it? 

REF-11) What outcome did you expect to result from your chosen COA? 

REF-11) What outcome actually occurred? 

REF-12) What alternative goals and problems did you identify in order to address the situation? 

REF-13) What alternative COA did you have to take in order to achieve the desired outcome? 
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Appendix F3. Sample reflection interventions: Reflection on Condition and Action 

Improving Problem Solving Skills (Battalion Commander) 

In this brief exercise, we would like to demonstrate how reflecting on the unexpected outcomes of our 
actions can provide information that improves our problem-solving capability. On the next few pages, 
you will be presented with the answers that other military personnel gave to one of the vignettes and 
the case study you just completed. You will be asked to reflect on the differences between your 
answers and the answers provided by others and to reflect on how these differences might have come 
about. 

One way to reflect on how differences in actions come about is to ask questions about where a 
person’s decisions to act come from. As you well know, different people can encounter the same 
situation but interpret it very differently. These different people therefore act very differently. With the 
questions we ask in this exercise, we are trying to turn your attention to how you come to understand 
the leadership problems you face on the job and how your understanding affects your decision to act. 
These questions may seem abstract, so they are explained below. The first two questions are: 

“What is the problem that must be solved? What goal do you intend to reach by solving it?” 

In these questions we are asking you to differentiate between a problem and a goal . A problem is a 
conflict that must be resolved, a challenge that must be overcome, or a balance that must be achieved 
in order to accomplish a particular goal . Using terms from the FM 101-5, a goal is analogous to a 
“mission” in that stating a goal involves stating the purpose for action. A problem is analogous to a 
“key task” in that problems represent the conditions that must be met for mission success or goal 
accomplishment. So, when we ask you to identify the problem in a vignette or case study, we are not 
asking you to provide us with a goal. We are asking you to state your goal and to identify what 
difficult thing must be accomplished in order to meet that goal. 

Consider the example below. What is the problem that must be solved in this situation? 

You are a battalion commander and it is the end of your first battle at a major externally-evaluated 
training exercise, during which your unit revealed some major shortcomings. During the AAR, the 
Chief Evaluator is highly critical of the battalion and dwells on all the negative things your unit did 
that day. You carefully record all of the negative observations, but you know full well that the 
battalion also did some very positive things that day. What should you do? 

Your goal in this situation might be to make sure that the Chief Evaluator recognizes the positive 
things your battalion did in the training exercise. In this case, the problem that must be solved is to 
find a way to assert yourself with the Chief Evaluator without undermining your credibility. 

An alternative goal might be to improve your battalion’s performance in future training exercises. In 
this case, the problem that must be solved is to establish and implement a set of methods that will lead 
to increased unit effectiveness and readiness. 

Another question you will see often is the following: 
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“Describe the factors you considered when choosing your goal and determining the problem that must 
be solved (e.g., doctrine, personal values, assumptions about army culture, procedures, and personnel, 
knowledge based on previous experiences)" 

By asking you to describe the factors listed above, we are trying to get you thinking about how you 
determine your goals and identify which problems to solve on the job. There are multiple ways to 
decide on goals and identify the critical problems to solve. These ways differ in how effectively they 
lead to a problem solution. By understanding how you choose a particular goal and how you come to 
identify the problems you face, you can better understand why you take the actions you do, and 
perhaps improve the effectiveness of your problem solving. 

Using the example above, if you defined the problem as one of asserting yourself with the Chief 
Evaluator, you might describe the factors as follows: 

“I assumed that the Chief Evaluator’s on-sided evaluation would have a negative effect on unit morale. 
I know from past experience that bad morale undermines unit cohesion and leadership capability. 

Also, I assumed that by asserting myself I would show my unit that I recognize their strengths. This 
would also improve morale.” 

If you defined the problem as one of establishing and implementing methods for increasing unit 
effectiveness and readiness, you might describe the factors as follows: 

“I assumed that the Chief Evaluator just had a negative style and that my unit would not take the on- 
sidedness of the evaluation too seriously. I have learned from past experience that you learn a lot more 
from hearing about your mistakes than from hearing praise. Evaluations are not about building self¬ 
esteem.” 

We understand that you did not necessarily think this analytically when you examined the vignettes 
and case study earlier in today’s session. Much of this kind of thinking is automatic and outside of our 
awareness. However, just as a golfer or baseball player must examine his swing in detail in order to 
improve it, we suspect that a leader must examine his reasons and thinking behind problem solving in 
order to improve his decision-making capability. 

On the following three pages, you will encounter the questions described above and you will be asked 
to reflect on how having different goals and identifying different problems lead to different problem¬ 
solving strategies. Please answer these questions thoughtfully and thoroughly, using the digital 
recorders. You will have approximately 40 minutes to complete the exercise. If at any point during this 
exercise you have questions about what we are asking you to do, please raise your hand and someone 
will assist you. 

Think about the first vignette you read. 

It is reprinted below with the average ratings given by 59 AWC students designated as expert battalion 
commanders. 
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B3. You are a new battalion commander and one of your most important and challenging tasks is to 
establish the training priorities for your unit. While everything looks important and you would like to 
meet every possible contingency, you also realize that you do not have the time or resources to “do it 
* all.” Rate the following strategies for how effective they would be in helping you establish your 
priorities. 

_8_Study the brigade's training schedule. 

_8_Talk to the brigade S-2, S-3, and CSM to verify your understanding of the brigade commander’s 

training focus. 

_7_Schedule meetings to discuss training with each of your staff members during your first week 

of command. 

_8_Explain your goals and your plans for the battalion very clearly to your officers and staff. 

4 Assess the tactical and technical competence of your Soldiers individually by giving them 
formal and informal tests. 

_6_Rely on the assessments made by the previous battalion commander. 

_7_Select three to five upcoming missions (based on the brigade training plan) to focus your 

Soldiers' energy on. 

_8_Before doing anything, make sure you understand the commander's intent two levels up. 

_8_Soon after taking command, visit each staff section's shop and get a full briefing on their 

operations. 

_8_Talk to the brigade commander to determine his training priorities. 

1 23 4 5 6 78 9 

_I_I_I_I_1_I_I_I_I_ 

Extremely Somewhat Neither Bad Somewhat Extremely 

Bad Bad Nor Good Good Good 

Note: See Part II for Overview and Instructions on Vignettes 

In this vignette, my ratings differed from the AWC students’ on the following options (please indicate 
with a check in the space provided on the left of the option): 

_Study the brigade's training schedule. _ 

_Talk to the brigade S-2, S-3, and CSM ... _ 

_Schedule meetings to discuss training ... _ 
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Explain your goals and your plans ... 


_Assess the tactical and technical competence ... _ 

_Rely on the assessments made ... _ 

_Select three to five upcoming missions ... _ 

_Before doing anything, make sure you understand ..._ 

_Soon after taking command, visit... _ 

_Talk to the brigade commander ... _ 

My ratings differed from the AWC students’ ratings in the following ways (Please indicate the degree 
of difference by writing the appropriate abbreviation to the right of the relevant response option.): 

(MH) Much higher (5 or more points difference) 

(H) Higher (3-4 points difference) 

(SH) Slightly higher (1-2 points difference) 

(SL) Slightly lower (1-2 points difference) 

(L) Lower (3-4 points difference) 

(ML) Much lower (5 or more points difference) 

My ratings differed from the students’ ratings perhaps because we had different goals in mind, and 
identified a different problem to be solved... 

CREF-la) What do you think is the problem that must be solved in this vignette? 

CREF-lb) What goal do you intend to reach by solving it? 

CREF-2a) What do you think the AWC students’ goal might have been? 

CREF-2b) What problem do you think they intended to solve? 

Differences in problem interpretation can happen for several reasons ... 

CREF-4) Describe the factors you considered when choosing your goal and determining the problem 
that must be solved, (e.g., doctrine, personal values, assumptions about army culture, procedures, and 
personnel, knowledge based on previous experiences) 

CREF-5) What factors do you imagine the AWC students considered? 


Think about the leadership problems in the case study you read. 

(You may flip back to it to review your answers, if necessary.) 

One of the questions from the case study is reprinted below with the average answer from 23 NCOs. 


74 






What CO A would you take to solve the problem? 

Approach CPT Powers about coming directly to me with his concerns. Call a meeting with PSG 
Newell and counsel him to ‘crack the whip’ with the squad leaders. Take a direct role in supervising 
training exercises. 

CREF-6) How does your COA differ from the one preferred by the NCOs? 

CREF-7) What problem do you suppose the NCOs identified as most important in order to prefer the 
COA they did? 

CREF-8) What factors did you consider when you chose your goal and identified the main problem in 
the case study? (e.g., doctrine, personal values, assumptions about army culture, procedures, and 
personnel, knowledge based on previous experiences) 


CREF-9) What factors do you imagine the NCOs considered? 


Think about a leadership problem that challenged you to re-examine your goals and assumptions, 
where your actions didn ’t result in the outcome you expected. 

CREF-10) Briefly describe the problem situation. What was your initial goal, and the problem you had 
to solve to achieve it? 

CREF-11) What outcome did you expect to result from your chosen COA? 

CREF-11) What outcome actually occurred? 

CREF-12) Describe the factors that played a role in choosing your goal and determining your problem 
identification, (e.g., doctrine, personal values, assumptions about army culture, procedures, and 
personnel, knowledge based on previous experiences) 

CREF-13) What factors did you have to consider to change your understanding of the problem? 
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Appendix F 4. Sample reflection interventions: College life (experimental) 


Reflection on Condition and Action: Improving Practical Problem Solving 

In this brief exercise we want to show you how reflecting on your thought process when confronting a 
practical problem can improve your ability to find the right solution. As you well know, different 
people can encounter the same situation but interpret it very differently. When we respond to 
situations in life, much of our thinking is automatic and outside of our awareness. However, just as a 
golfer or baseball player examines his or her swing in detail in order to improve performance, 
examining thinking that underlies problem solving makes it possible to improve decision-making. 

On the next few pages you will be asked to reflect on one of the college life vignettes and an 
experience of your own. You will be guided through a series of questions that turn your attention to 
how you came to understand the problems presented and how this understanding affected your 
decision to act. You have probably heard the phrase “think before you act”. Our objective is to help 
you become aware of your thinking process and develop it. 

In particular, we will ask you to reflect on three fundamental components of problem solving: 1) what 
you define as a problem; 2) the goal or outcome you hope to achieve; and 3) what actions you expect 
will bring about your chosen outcome. 

To illustrate these points, consider the vignette below. 

One evening, you come to the dining hall and attempt to join the crowd of friends you usually eat 
with. You get your food, you approach the table, you are about to say “Hi, guys!” and, all of a sudden, 
you notice that nobody greets you and nobody smiles at you. Quite on the contrary, some people are 
looking down while others are just staring at you. 

There are several feasible interpretations of this problem. Quite often the way we interpret a problem 
is directly linked to a past experience that seems similar in certain respects. Some examples are as 
follows: 

1) You may interpret the problem to be that your friends are upset with you about something you have 
done. Perhaps you have experienced a similar situation in high school when a few of your good 
friends were angry with you and chose to ignore you. 

2) Alternatively this situation may remind you of a time when, for no apparent reason at all, certain 
friends turned against you. This experience may lead you interpret the current problem in a similar 
fashion. 

3) A third interpretation may be that, if there is a problem, it does not necessarily involve you. Perhaps 
you have encountered a situation like this before when you needlessly worried about a problem that 
didn’t involve you in the first place. 

Therefore, it can be helpful to become aware of how our past experience may influence and sometimes 
bias how we interpret a new situation. To avoid ways that our past experience may limit our 
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perspective, it is important to pay attention to unique factors in the new situation and consider 
alternative interpretations before responding. 

In the above example, a course of action that you may choose to take given your particular 
interpretation of a situation depends upon your goal or the outcome you wish to achieve. 

For example: 

1) If you interpret the problem to be that you may have done something wrong, a goal or intended 
outcome might be to restore your friendship. This could lead to actions associated with finding out 
what’s wrong so you can do something to straighten it out. 

2) If you feel these friends are mistreating you, your goal from this perspective might be to protect 
yourself from being hurt. This might lead to actions associated with avoiding them. 

3) If you feel that whatever is going on probably does not involve you your goal could be to avoid the 
problem entirely. This could lead to actions associated with pretending that there is nothing wrong 
and going about your own business. 

Just as there are several ways to interpret problems that are reflected in the goals that you chose there 
are also several ways to reach those goals. After you have defined your problem, it is important to 
consider alternative actions before deciding on the best course of action. For example, an alternate 
approach to dealing with the problem of being mistreated by friends with the goal of protecting 
yourself could be to confront the issue and defend yourself. 

We can improve our practical problem solving by recognizing what factors lead us to interpret a 
situation in a particular way, form our goals, and select actions to achieve them. 

Now consider another vignette that you responded to earlier, which is reprinted with response options 
below to refresh your memory. 

You have decided to apply for an internship during the upcoming break, and want to ask one of your 
professors for a letter of recommendation. The professor you have in mind is teaching a fairly large 
class, and he does not know you very well. One day you run in to him in the coffee shop, where he is 
sitting with what you assume are his kids 

a) You decide that this is a good time to talk to him about the letter. 

b) You go up and greet him, reminding him of your name and what class you are in. 

c) You greet him and then start chatting with his kids. 

d) You nod but do not talk to him. 

e) You pretend you have not seen him. He probably does not want to deal with students outside of his 
workplace. 

f) You ask if you can sit down with him and his kids and talk about different things. 

g) You greet him and ask for an appointment with him the following day. 

h) You greet him and offer to buy him and his kids coffee or sodas. 

Please respond to the following questions as if you are the student in this vignette: 
la) What is your interpretation of the problem to be solved in this vignette? 
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lb) What is the goal or outcome you are trying to achieve? 

lc) What specific course of action do you think would be most useful to achieving your goal? 

l d) How would you know if this course of action was NOT effective? 

le) What factors (e.g., past experience with professors, beliefs, values, etc.) do you think influenced 
how you interpreted the situation and selected your goal and preferred course of action? 

l f) Think about the perspective of the faculty member. How might he interpret the situation? 

2a) Suggest an interpretation of the problem in this vignette that is feasible but different than yours. 
2b) What factors would you have to emphasize to arrive at this interpretation? 

2c) Suggest a goal that might be associated with this alternative interpretation that is feasible but 
different than your original goal. 

2d) Suggest a course of action that you think might be effective given this new interpretation and goal. 

Now, please think about a situation you encountered in either high school or college in which your 
actions did NOT result in the outcome that you expected and you were challenged to reexamine your 
assumptions, goals, and/or actions. 

3a) Briefly describe the problem situation. What was your interpretation of the problem, the goal you 
had in mind, and the action you took? 

3b) Describe the factors (e.g., assumptions, beliefs, values, past experience, etc.) that played a role in 
how you defined the problem, selected your goal, and/or the action you took. 

3c) What outcome did you expect would result from the action that you took? 

3d) What was the actual outcome? 

3e) What factors did you need to reconsider to change your understanding of the problem? 

3f) In light of this new understanding, what would be your goal and how would you act differently in a 
situation such as this? 

Recognizing that the factors we consider and emphasize play an important part in determining how we 
interpret problem situations: 

Please list at least three questions that you can ask yourself before responding to a problem situation. 

Realizing that our choice of actions may be biased by assumptions, beliefs, and past experience: 

Please list at least three questions that you can ask yourself before deciding on the particular course of 
action to take in a situation 

We believe that approaching practical problems in this way can help you improve your capacity to 
solve problems effectively! 
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Appendix F 4. Sample reflection interventions: College life (control) 

Please read the following article and respond to the questions provided. 

Living with the Opposite Sex - In vour room 

Adapted from an article by BROCK MCCORMACK 
STUDENT.COM STAFF WRITER 

Imagine for a moment that you're in a dorm bathroom, just stepping out of the shower. You adjust 
your towel to cover the necessary parts, and as you look up, the cutie down the hall is standing inches 
from you. You've been turned on by your neighbor for months, and now you've finally met — in the 
bathroom. 

Increasingly, schools are allowing male and female students to share bathrooms, suites, and in some 
cases, individual rooms. If you struggle to resist sexual urges on a club dance floor, or at a party, 
imagine sharing a room with someone of the opposite sex. Schools like Haverford College in 
Pennsylvania, Wesleyan University in Connecticut, and Hampshire College in Massachusetts have 
already adopted liberal housing policies, whereas Tufts University, located outside Boston, recently 
rejected the idea. 

Nick D'Avella, a senior at Haverford, says, "I've been sharing a bathroom with women since my 
freshman year, and now I am living in an apartment with a woman. It's so not a big deal; it's 
ridiculous. I'm really surprised everyone outside the college (including a bunch of alums) seem to see 
this as an easy way for college kids to have sex or something." 

Co-ed living forces everyone to be sensitive to hygiene and cleanliness issues, as the presence of a 
mere bathtub hairball can spark a bathroom war. Junior Thea Pratt at Wesleyan recalls how her dorm 
floor had both a co-ed bathroom and single-sex facilities, and boys were requested to use the urinal 
only in their bathroom. Some even use special signals to request privacy in the bathroom, like a secret 
door handle decoration or magnet. 

Signs are useful for reminding suitemates to remove hair from the drain, or wipe toothpaste from the 
sink, although they can seem immature. Haverford senior Erin Armstrong notes that some suites at her 
school have had locks installed, an easy way to create a sense of privacy. 

The obvious drawback of co-ed living arrangements is that some students will inevitably choose to 
live with a significant other, leaving residence administrators to sort out bitter break-ups. Haverford 
junior Rob Barry thinks, "If people are stupid enough to take advantage of this freedom by living with 
a significant other, then frankly it's better if they make the mistake while still in college rather than out 
in the real world." Living with a significant other might be great at the outset, but once an argument 
erupts, coming home will seem like a nightmare. 

D'Avella knew some couples whose relationship soured mid-year, and says, "Some of them deal with 
it fine (the breakup — the actual dating isn't really a problem) and for some it's messy. You just deal 
with it. Kind of like if you suddenly start hating your best friend that you live with I guess...just work 
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something out and get through the year." Unless the residential life department ships one of you out, 
it's probably best to just confront the situation and work out a truce. 

Wesleyan has a decidedly mature approach to its co-ed policy, essentially disclaiming responsibility 
for sorting out students' bad decisions. The school's residential life office says bluntly that "a male and 
female student can choose to live together," implying that the school really doesn't mind either way. 

Anyone considering living in a co-ed room should first weigh the problems that might arise, and 
whether you are ready to put up with annoying bathroom or bedroom habits. Living with the opposite 
sex is usually great, just as long as you use some common sense. 

Brock McCormack lives with two women, and he loves it. 

Questions: 

1) Do you think it’s wise for colleges to have co-ed rooms? Why or why not? 

2) What policies do you think colleges should have concerning co-ed living? 

3) How would you feel about sharing a room with a co-ed? 

4) How is living with the same sex different from living with the opposite sex? 

5) What advantages and disadvantages do you see to having co-eds share 

bathrooms? Suites? 

Advantages: 

Disadvantages: 

6) Twenty years ago, many colleges kept male and female students in separate buildings with 
strict visiting policies. Why do you think these rules existed at this time 

7) What do you think has led to the shift in attitudes regarding co-ed living arrangements? 

8) In general, how do you feel about our society’s approach towards relationships between men 
and women as a whole? 

9) How have your past experiences with roommates influenced your answers to the previous 
questions? 
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